Aug 04 00:00:29.359091 ip-10-0-138-102 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Aug 04 00:00:29.359101 ip-10-0-138-102 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Aug 04 00:00:29.359110 ip-10-0-138-102 systemd[1]: kubelet.service: Failed with result 'resources'. Aug 04 00:00:29.359480 ip-10-0-138-102 systemd[1]: Failed to start Kubernetes Kubelet. Aug 04 00:00:39.530581 ip-10-0-138-102 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Aug 04 00:00:39.530602 ip-10-0-138-102 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot dba3296cfea942e18521c35c3cdf38eb -- Aug 04 00:02:31.484376 ip-10-0-138-102 systemd[1]: Starting Kubernetes Kubelet... Aug 04 00:02:31.900593 ip-10-0-138-102 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 00:02:31.900593 ip-10-0-138-102 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 00:02:31.900593 ip-10-0-138-102 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 00:02:31.900593 ip-10-0-138-102 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 00:02:31.900593 ip-10-0-138-102 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 00:02:31.901541 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.901461 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 00:02:31.905899 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905885 2570 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:31.905899 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905898 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:31.905899 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905901 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905904 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905908 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905911 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905914 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905917 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905919 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905922 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905925 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905928 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905930 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905933 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905935 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905938 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905941 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905943 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905945 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905948 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905951 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905954 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:31.905985 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905957 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905959 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905962 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905964 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905967 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905970 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905973 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905975 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905978 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905980 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905982 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905985 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905987 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905990 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905993 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.905998 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906002 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906005 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906008 2570 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906011 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:31.906460 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906013 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906016 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906019 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906021 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906024 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906028 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906031 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906034 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906037 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906040 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906043 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906045 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906048 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906051 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906054 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906057 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906059 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906062 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906064 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:31.907261 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906067 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906069 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906072 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906074 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906077 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906079 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906081 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906085 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906089 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906092 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906096 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906098 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906101 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906104 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906107 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906109 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906112 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906114 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906117 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:31.907803 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906132 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906134 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906137 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906140 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906142 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906145 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906147 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906150 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906154 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906157 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906159 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906162 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906164 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906167 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906169 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906172 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906174 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906177 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:31.908278 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.906179 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:31.910590 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910575 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:31.910590 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910589 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:31.910590 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910593 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910596 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910599 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910602 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910605 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910607 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910611 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910613 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910616 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910619 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910621 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910624 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910627 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910629 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910632 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910635 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910637 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910640 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910642 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910645 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:31.910696 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910648 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910650 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910653 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910656 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910659 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910663 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910667 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910670 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910673 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910675 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910678 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910680 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910683 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910685 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910690 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910693 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910695 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910699 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910701 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910704 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:31.911213 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910708 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910710 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910713 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910716 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910718 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910722 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910727 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910730 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910734 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910736 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910741 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910744 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910747 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910749 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910752 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910755 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910758 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910761 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910764 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:31.911709 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910767 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910770 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910772 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910774 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910777 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910780 2570 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910782 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910785 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910788 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910791 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910793 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910796 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910799 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910801 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910804 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910807 2570 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910809 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910812 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910814 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910817 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:31.912198 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910820 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910822 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910825 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910828 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910830 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910833 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910835 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910838 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910840 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910843 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910846 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910848 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910851 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910853 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910856 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910859 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910862 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.910864 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911243 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:31.912698 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911248 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911251 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911254 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911256 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911259 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911262 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911265 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911268 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911270 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911273 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911275 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911278 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911281 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911284 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911286 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911289 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911292 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911295 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911298 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:31.913173 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911300 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911303 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911305 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911308 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911311 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911313 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911315 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911318 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911320 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911323 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911325 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911328 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911330 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911333 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911336 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911338 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911367 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911370 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911373 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911376 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:31.913627 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911378 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911381 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911383 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911386 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911388 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911391 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911393 2570 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911396 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911398 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911401 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911404 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911406 2570 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911409 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911414 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911418 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911422 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911425 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911428 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911430 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911432 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:31.914171 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911435 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911437 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911440 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911442 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911445 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911448 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911451 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911453 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911456 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911458 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911461 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911464 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911466 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911469 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911471 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911474 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911476 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911479 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911482 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911484 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:31.914647 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911487 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911489 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911493 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911498 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911501 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911504 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911506 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911509 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911513 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911516 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911519 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911522 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911525 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911528 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911530 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911533 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911536 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911539 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.911541 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911609 2570 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 00:02:31.915108 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911616 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911622 2570 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911626 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911631 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911634 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911638 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911641 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911645 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911648 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911651 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911654 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911657 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911660 2570 flags.go:64] FLAG: --cgroup-root="" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911664 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911666 2570 flags.go:64] FLAG: --client-ca-file="" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911671 2570 flags.go:64] FLAG: --cloud-provider="external" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911674 2570 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911678 2570 flags.go:64] FLAG: --cluster-domain="" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911681 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911684 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911687 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911690 2570 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911693 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 00:02:31.915623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911696 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911699 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911703 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911706 2570 flags.go:64] FLAG: --contention-profiling="false" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911708 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911712 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911715 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911718 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911722 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911725 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911729 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911731 2570 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911734 2570 flags.go:64] FLAG: --enable-server="true" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911737 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911742 2570 flags.go:64] FLAG: --event-burst="100" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911745 2570 flags.go:64] FLAG: --event-qps="50" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911748 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911751 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911753 2570 flags.go:64] FLAG: --eviction-hard="" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911757 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911760 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911763 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911766 2570 flags.go:64] FLAG: --eviction-soft="" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911769 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911773 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 00:02:31.916286 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911776 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911779 2570 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911782 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911785 2570 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911788 2570 flags.go:64] FLAG: --feature-gates="" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911791 2570 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911794 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911797 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911800 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911803 2570 flags.go:64] FLAG: --healthz-port="10248" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911806 2570 flags.go:64] FLAG: --help="false" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911809 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-138-102.ec2.internal" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911814 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911816 2570 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911819 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911823 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911826 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911829 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911832 2570 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911835 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911838 2570 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911840 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911844 2570 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911847 2570 flags.go:64] FLAG: --kube-reserved="" Aug 04 00:02:31.916919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911850 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911852 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911855 2570 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911858 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911861 2570 flags.go:64] FLAG: --lock-file="" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911864 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911866 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911869 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911874 2570 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911879 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911882 2570 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911884 2570 flags.go:64] FLAG: --logging-format="text" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911887 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911891 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911894 2570 flags.go:64] FLAG: --manifest-url="" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911896 2570 flags.go:64] FLAG: --manifest-url-header="" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911900 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911903 2570 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911907 2570 flags.go:64] FLAG: --max-pods="110" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911910 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911914 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911917 2570 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911920 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911923 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911926 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 00:02:31.917511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911929 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911935 2570 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911938 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911942 2570 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911945 2570 flags.go:64] FLAG: --pod-cidr="" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911948 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911953 2570 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911956 2570 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911959 2570 flags.go:64] FLAG: --pods-per-core="0" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911962 2570 flags.go:64] FLAG: --port="10250" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911965 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911968 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07b1646500a80c9c0" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911971 2570 flags.go:64] FLAG: --qos-reserved="" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911974 2570 flags.go:64] FLAG: --read-only-port="10255" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911977 2570 flags.go:64] FLAG: --register-node="true" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911980 2570 flags.go:64] FLAG: --register-with-taints="" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911984 2570 flags.go:64] FLAG: --registry-burst="10" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911990 2570 flags.go:64] FLAG: --registry-qps="5" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911993 2570 flags.go:64] FLAG: --reserved-cpus="" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911995 2570 flags.go:64] FLAG: --reserved-memory="" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.911999 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912002 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912005 2570 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912008 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912010 2570 flags.go:64] FLAG: --runonce="false" Aug 04 00:02:31.918512 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912013 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912016 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912019 2570 flags.go:64] FLAG: --seccomp-default="false" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912025 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912027 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912031 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912034 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912037 2570 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912039 2570 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912042 2570 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912045 2570 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912048 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912051 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912054 2570 flags.go:64] FLAG: --system-cgroups="" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912056 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912062 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912065 2570 flags.go:64] FLAG: --tls-cert-file="" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912068 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912071 2570 flags.go:64] FLAG: --tls-min-version="" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912074 2570 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912076 2570 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912079 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912082 2570 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912085 2570 flags.go:64] FLAG: --v="2" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912089 2570 flags.go:64] FLAG: --version="false" Aug 04 00:02:31.919603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912093 2570 flags.go:64] FLAG: --vmodule="" Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912096 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912099 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912219 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912223 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912227 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912230 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912233 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912236 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912239 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912244 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912247 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912250 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912253 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912255 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912259 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912263 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912268 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912271 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:31.920720 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912275 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912278 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912281 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912284 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912286 2570 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912289 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912292 2570 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912294 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912297 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912299 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912302 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912304 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912307 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912311 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912315 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912318 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912321 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912323 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912326 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912329 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:31.921455 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912332 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912334 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912337 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912341 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912344 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912347 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912350 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912352 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912355 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912358 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912360 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912363 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912367 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912370 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912372 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912375 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912378 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912380 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912383 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912386 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:31.922077 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912389 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912391 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912394 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912396 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912399 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912401 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912404 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912407 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912410 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912413 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912415 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912418 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912420 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912423 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912425 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912430 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912432 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912435 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912438 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912440 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:31.922920 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912443 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912445 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912448 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912451 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912454 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912457 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912460 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912463 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912466 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912468 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912471 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912473 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912476 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912478 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912481 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912484 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912487 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912489 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912492 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912494 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:31.923563 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912497 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:31.924224 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912500 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:31.924224 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.912503 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:31.924224 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.912953 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 00:02:31.924224 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.919762 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 00:02:31.924224 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.919781 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 00:02:31.924224 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.919808 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 00:02:31.924224 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.919816 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 00:02:31.924224 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919894 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:31.924224 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919901 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:31.924224 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919907 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:31.924224 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919911 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:31.924224 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919916 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:31.924224 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919921 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:31.924224 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919925 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:31.924224 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919929 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919933 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919940 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919946 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919951 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919955 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919959 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919964 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919969 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919973 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919978 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919983 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919987 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919991 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919995 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.919999 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920003 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920009 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920013 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920018 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:31.924626 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920022 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920026 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920031 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920036 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920040 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920044 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920049 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920053 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920060 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920066 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920071 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920076 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920080 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920085 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920089 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920093 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920098 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920102 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920106 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:31.925093 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920111 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920115 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920139 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920144 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920148 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920153 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920157 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920161 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920166 2570 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920170 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920174 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920180 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920185 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920189 2570 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920193 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920198 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920203 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920208 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920212 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920216 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:31.925602 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920221 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920224 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920229 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920233 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920237 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920243 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920249 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920254 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920259 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920263 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920268 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920272 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920277 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920281 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920285 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920289 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920294 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920298 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920302 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:31.926105 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920306 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:31.926615 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920310 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:31.926615 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920314 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:31.926615 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920319 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:31.926615 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920323 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:31.926615 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920329 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:31.926615 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920333 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:31.926615 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920337 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:31.926615 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920341 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:31.926615 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920346 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:31.926615 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920350 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:31.926615 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920354 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:31.926615 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920358 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:31.926615 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920362 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:31.926615 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.920370 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920540 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920547 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920552 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920556 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920561 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920565 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920570 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920574 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920578 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920583 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920587 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920591 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920595 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920600 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920604 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920608 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920612 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920616 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920621 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:31.926953 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920625 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920629 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920633 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920637 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920642 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920647 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920651 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920655 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920659 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920663 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920667 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920672 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920676 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920680 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920685 2570 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920691 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920697 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920701 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920705 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:31.927505 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920709 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920713 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920717 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920722 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920726 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920730 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920734 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920737 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920741 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920746 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920750 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920754 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920759 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920763 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920767 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920771 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920776 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920780 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920785 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920789 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:31.928066 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920793 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920797 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920804 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920809 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920814 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920819 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920823 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920828 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920832 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920836 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920841 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920847 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920853 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920858 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920862 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920867 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920871 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920875 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920880 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:31.928896 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920884 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920888 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920892 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920896 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920900 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920904 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920909 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920913 2570 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920917 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920921 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920925 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920929 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920934 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920938 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920942 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920946 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920950 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920954 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920958 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920962 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:31.929493 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920967 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:31.929971 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:31.920971 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:31.929971 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.920978 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 00:02:31.929971 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.921108 2570 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 00:02:31.929971 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.923770 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 00:02:31.929971 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.925539 2570 server.go:1019] "Starting client certificate rotation" Aug 04 00:02:31.929971 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.925639 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 00:02:31.929971 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.925676 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 00:02:31.929971 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.928444 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 00:02:31.931434 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.931417 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 00:02:31.942022 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.942004 2570 log.go:25] "Validated CRI v1 runtime API" Aug 04 00:02:31.943964 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.943947 2570 log.go:25] "Validated CRI v1 image API" Aug 04 00:02:31.945506 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.945492 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 00:02:31.948713 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.948696 2570 fs.go:135] Filesystem UUIDs: map[6f2e0ef0-bb61-4ab6-b1ae-135685b9a347:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c184cfbd-9ba3-4943-8a78-99199658c1d2:/dev/nvme0n1p4] Aug 04 00:02:31.948765 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.948713 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 00:02:31.954646 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.954556 2570 manager.go:216] Machine: {Timestamp:2026-08-04 00:02:31.953398718 +0000 UTC m=+0.352166210 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3111194 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec292da8b0b9c8f8852726560182ef5f SystemUUID:ec292da8-b0b9-c8f8-8527-26560182ef5f BootID:dba3296c-fea9-42e1-8521-c35c3cdf38eb Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a6:1d:35:08:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a6:1d:35:08:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:3a:41:f3:87:58 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 00:02:31.954646 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.954642 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 00:02:31.954743 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.954704 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 00:02:31.955024 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.955007 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 00:02:31.955767 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.955740 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 00:02:31.955885 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.955770 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-102.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 00:02:31.956396 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.956387 2570 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 00:02:31.956426 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.956404 2570 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 00:02:31.956426 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.956412 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 00:02:31.956979 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.956969 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 00:02:31.957646 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.957637 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 00:02:31.960464 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.960453 2570 state_mem.go:36] "Initialized new in-memory state store" Aug 04 00:02:31.960568 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.960559 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 00:02:31.960600 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.960588 2570 kubelet.go:486] "Attempting to sync node with API server" Aug 04 00:02:31.960633 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.960605 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 00:02:31.960633 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.960619 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 00:02:31.960633 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.960628 2570 kubelet.go:398] "Adding apiserver pod source" Aug 04 00:02:31.960705 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.960638 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 00:02:31.960892 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.960882 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 00:02:31.960930 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.960898 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 00:02:31.964267 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.964252 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 00:02:31.965452 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.965439 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 00:02:31.965493 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.965468 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 00:02:31.966593 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.966582 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 00:02:31.966626 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.966598 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 00:02:31.966626 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.966604 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 00:02:31.966626 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.966610 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 00:02:31.966626 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.966618 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 00:02:31.966626 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.966623 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 00:02:31.966752 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.966632 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 00:02:31.966752 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.966638 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 00:02:31.966752 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.966644 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 00:02:31.966752 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.966649 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 00:02:31.966752 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.966656 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 00:02:31.966752 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.966662 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 00:02:31.967270 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.967261 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 00:02:31.967301 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.967272 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 00:02:31.970258 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.970244 2570 server.go:1268] "Started kubelet" Aug 04 00:02:31.970388 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.970325 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 00:02:31.970496 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.970436 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 00:02:31.970496 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.970390 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 00:02:31.970906 ip-10-0-138-102 systemd[1]: Started Kubernetes Kubelet. Aug 04 00:02:31.971735 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.971719 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 00:02:31.975871 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.975846 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-102.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 00:02:31.976060 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:31.976029 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-102.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 00:02:31.976174 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:31.976069 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 00:02:31.977597 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:31.977577 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 00:02:31.978025 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.978014 2570 server.go:310] "Adding debug handlers to kubelet server" Aug 04 00:02:31.980053 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.980037 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 00:02:31.980151 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.980054 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 00:02:31.980151 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:31.980113 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found" Aug 04 00:02:31.980486 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.980468 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 00:02:31.980595 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.980468 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 00:02:31.980595 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.980515 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 00:02:31.980690 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.980627 2570 reconstruct.go:97] "Volume reconstruction finished" Aug 04 00:02:31.980690 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.980635 2570 reconciler.go:29] "Reconciler: start to sync state" Aug 04 00:02:31.980939 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.980923 2570 factory.go:55] Registering systemd factory Aug 04 00:02:31.981030 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.980961 2570 factory.go:223] Registration of the systemd container factory successfully Aug 04 00:02:31.981239 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.981222 2570 factory.go:153] Registering CRI-O factory Aug 04 00:02:31.981308 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.981243 2570 factory.go:223] Registration of the crio container factory successfully Aug 04 00:02:31.981308 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.981293 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 00:02:31.981397 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.981316 2570 factory.go:103] Registering Raw factory Aug 04 00:02:31.981397 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.981328 2570 manager.go:1196] Started watching for new ooms in manager Aug 04 00:02:31.981647 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.981636 2570 manager.go:349] Starting recovery of all containers Aug 04 00:02:31.983209 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:31.983047 2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-102.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 00:02:31.983288 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:31.983218 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 00:02:31.984206 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:31.983176 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-102.ec2.internal.18c8713b5974895f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-102.ec2.internal,UID:ip-10-0-138-102.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-102.ec2.internal,},FirstTimestamp:2026-08-04 00:02:31.970220383 +0000 UTC m=+0.368987874,LastTimestamp:2026-08-04 00:02:31.970220383 +0000 UTC m=+0.368987874,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-102.ec2.internal,}" Aug 04 00:02:31.992074 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.992060 2570 manager.go:354] Recovery completed Aug 04 00:02:31.993638 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:31.993613 2570 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Aug 04 00:02:31.998263 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.998247 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:31.999955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.999934 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:32.000028 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.999965 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:32.000028 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:31.999975 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:32.000449 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.000434 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 00:02:32.000449 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.000443 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:32.000449 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.000449 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 00:02:32.000613 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.000466 2570 state_mem.go:36] "Initialized new in-memory state store" Aug 04 00:02:32.001084 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.001070 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:32.001152 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.001096 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:32.001152 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.001111 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:32.001772 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.001759 2570 policy_none.go:49] "None policy: Start" Aug 04 00:02:32.001839 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.001792 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 00:02:32.001839 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.001806 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 00:02:32.002337 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:32.002272 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-102.ec2.internal.18c8713b5b3a34fd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-102.ec2.internal,UID:ip-10-0-138-102.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-102.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-102.ec2.internal,},FirstTimestamp:2026-08-04 00:02:31.999952125 +0000 UTC m=+0.398719618,LastTimestamp:2026-08-04 00:02:31.999952125 +0000 UTC m=+0.398719618,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-102.ec2.internal,}" Aug 04 00:02:32.002718 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.002706 2570 policy_none.go:47] "Start" Aug 04 00:02:32.003330 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.003316 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r8jvt" Aug 04 00:02:32.010023 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:32.009886 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-102.ec2.internal.18c8713b5b3a7c42 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-102.ec2.internal,UID:ip-10-0-138-102.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-102.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-102.ec2.internal,},FirstTimestamp:2026-08-04 00:02:31.99997037 +0000 UTC m=+0.398737860,LastTimestamp:2026-08-04 00:02:31.99997037 +0000 UTC m=+0.398737860,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-102.ec2.internal,}" Aug 04 00:02:32.011044 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.011029 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r8jvt" Aug 04 00:02:32.012621 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:32.012570 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-102.ec2.internal.18c8713b5b3a9ff5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-102.ec2.internal,UID:ip-10-0-138-102.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-102.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-102.ec2.internal,},FirstTimestamp:2026-08-04 00:02:31.999979509 +0000 UTC m=+0.398746999,LastTimestamp:2026-08-04 00:02:31.999979509 +0000 UTC m=+0.398746999,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-102.ec2.internal,}" Aug 04 00:02:32.044642 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.044622 2570 manager.go:337] "Starting Device Plugin manager" Aug 04 00:02:32.044716 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:32.044653 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 00:02:32.044716 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.044662 2570 server.go:87] "Starting device plugin registration server" Aug 04 00:02:32.044841 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.044826 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 00:02:32.044841 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.044839 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 00:02:32.045002 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.044925 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 00:02:32.045002 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.044997 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 00:02:32.045082 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.045006 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 00:02:32.045475 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:32.045412 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 00:02:32.045475 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:32.045460 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-102.ec2.internal\" not found" Aug 04 00:02:32.109870 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.109848 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 00:02:32.111000 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.110984 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 00:02:32.111055 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.111027 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 00:02:32.111055 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.111047 2570 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 00:02:32.111160 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:32.111082 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 00:02:32.113523 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.113501 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 00:02:32.145630 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.145612 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:32.146322 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.146308 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:32.146392 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.146332 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:32.146392 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.146341 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:32.146392 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.146360 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-102.ec2.internal" Aug 04 00:02:32.156189 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.156150 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-102.ec2.internal" Aug 04 00:02:32.156189 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:32.156168 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-102.ec2.internal\": node \"ip-10-0-138-102.ec2.internal\" not found" Aug 04 00:02:32.184831 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:32.184813 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found" Aug 04 00:02:32.211657 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.211637 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal"] Aug 04 00:02:32.211740 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.211695 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:34.128819 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.212414 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:34.128819 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.212437 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:34.128819 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:32.212453 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:34.129907 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.129895 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:34.130028 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.130013 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal" Aug 04 00:02:34.130088 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.130039 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:34.130559 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.130544 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:34.130633 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.130566 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:34.130633 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.130582 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:34.130708 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.130640 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:34.130708 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.130666 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:34.130708 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.130677 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:34.131817 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.131804 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal" Aug 04 00:02:34.131869 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.131825 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:34.132432 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.132412 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:34.132520 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.132441 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:34.132520 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.132451 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:34.173754 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:34.173737 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-102.ec2.internal\" not found" node="ip-10-0-138-102.ec2.internal" Aug 04 00:02:34.177859 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:34.177843 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-102.ec2.internal\" not found" node="ip-10-0-138-102.ec2.internal" Aug 04 00:02:34.198795 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.198756 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/470b6ba76b1c422c0a4df3ff57e9cabb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal\" (UID: \"470b6ba76b1c422c0a4df3ff57e9cabb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal" Aug 04 00:02:34.198876 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.198800 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/470b6ba76b1c422c0a4df3ff57e9cabb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal\" (UID: \"470b6ba76b1c422c0a4df3ff57e9cabb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal" Aug 04 00:02:34.198876 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.198816 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da457b2ccc5cb1e2a6488c095c0a0489-config\") pod \"kube-apiserver-proxy-ip-10-0-138-102.ec2.internal\" (UID: \"da457b2ccc5cb1e2a6488c095c0a0489\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal" Aug 04 00:02:34.201284 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:34.201273 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found" Aug 04 00:02:34.299865 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.299849 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/470b6ba76b1c422c0a4df3ff57e9cabb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal\" (UID: \"470b6ba76b1c422c0a4df3ff57e9cabb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal" Aug 04 00:02:34.299919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.299904 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/470b6ba76b1c422c0a4df3ff57e9cabb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal\" (UID: \"470b6ba76b1c422c0a4df3ff57e9cabb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal" Aug 04 00:02:34.299959 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.299870 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/470b6ba76b1c422c0a4df3ff57e9cabb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal\" (UID: \"470b6ba76b1c422c0a4df3ff57e9cabb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal" Aug 04 00:02:34.299959 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.299943 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/470b6ba76b1c422c0a4df3ff57e9cabb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal\" (UID: \"470b6ba76b1c422c0a4df3ff57e9cabb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal" Aug 04 00:02:34.300029 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.299971 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da457b2ccc5cb1e2a6488c095c0a0489-config\") pod \"kube-apiserver-proxy-ip-10-0-138-102.ec2.internal\" (UID: \"da457b2ccc5cb1e2a6488c095c0a0489\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal" Aug 04 00:02:34.300029 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.300003 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da457b2ccc5cb1e2a6488c095c0a0489-config\") pod \"kube-apiserver-proxy-ip-10-0-138-102.ec2.internal\" (UID: \"da457b2ccc5cb1e2a6488c095c0a0489\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal" Aug 04 00:02:34.301972 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:34.301957 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found" Aug 04 00:02:34.402834 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:34.402772 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found" Aug 04 00:02:34.476971 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.476950 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal" Aug 04 00:02:34.481102 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.480854 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal" Aug 04 00:02:34.503055 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:34.503036 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found" Aug 04 00:02:34.603539 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:34.603512 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found" Aug 04 00:02:34.704027 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:34.703981 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found" Aug 04 00:02:34.804395 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:34.804374 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found" Aug 04 00:02:34.840551 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.840530 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 00:02:34.840924 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.840678 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 00:02:34.905151 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:34.905133 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found" Aug 04 00:02:34.931015 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.930986 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-02 23:57:33 +0000 UTC" deadline="2028-02-09 12:15:48.104403699 +0000 UTC" Aug 04 00:02:34.931128 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.931018 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13308h13m13.17339298s" Aug 04 00:02:34.956122 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:34.954879 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 00:02:35.004608 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:35.004574 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod470b6ba76b1c422c0a4df3ff57e9cabb.slice/crio-6a47efd555a1e460fd94eb212db98008355f840b745460d7d80b188e614b1dc2 WatchSource:0}: Error finding container 6a47efd555a1e460fd94eb212db98008355f840b745460d7d80b188e614b1dc2: Status 404 returned error can't find the container with id 6a47efd555a1e460fd94eb212db98008355f840b745460d7d80b188e614b1dc2 Aug 04 00:02:35.005034 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:35.005015 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda457b2ccc5cb1e2a6488c095c0a0489.slice/crio-e408982b35f9f00b1c368c1d199d0ca8dcdca774e9a5ac1455110489ff00f9c2 WatchSource:0}: Error finding container e408982b35f9f00b1c368c1d199d0ca8dcdca774e9a5ac1455110489ff00f9c2: Status 404 returned error can't find the container with id e408982b35f9f00b1c368c1d199d0ca8dcdca774e9a5ac1455110489ff00f9c2 Aug 04 00:02:35.006101 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:35.006081 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found" Aug 04 00:02:35.010248 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.010190 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 00:02:35.030026 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.029989 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal" event={"ID":"470b6ba76b1c422c0a4df3ff57e9cabb","Type":"ContainerStarted","Data":"6a47efd555a1e460fd94eb212db98008355f840b745460d7d80b188e614b1dc2"} Aug 04 00:02:35.030920 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.030902 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal" event={"ID":"da457b2ccc5cb1e2a6488c095c0a0489","Type":"ContainerStarted","Data":"e408982b35f9f00b1c368c1d199d0ca8dcdca774e9a5ac1455110489ff00f9c2"} Aug 04 00:02:35.066062 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.066043 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:35.066871 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.066858 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:35.066944 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.066884 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:35.066944 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.066900 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:35.107096 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:35.107069 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found" Aug 04 00:02:35.207593 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:35.207541 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found" Aug 04 00:02:35.308054 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:35.308031 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found" Aug 04 00:02:35.394869 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.394846 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 00:02:35.408979 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:35.408956 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-102.ec2.internal\" not found" Aug 04 00:02:35.442385 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.442367 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 00:02:35.497342 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.497287 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal" Aug 04 00:02:35.509960 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.509941 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 00:02:35.512190 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.512173 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal" Aug 04 00:02:35.518781 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.518765 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 00:02:35.878438 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.878385 2570 apiserver.go:52] "Watching apiserver" Aug 04 00:02:35.885765 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.885746 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 00:02:35.886934 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.886912 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-cqddv","openshift-network-operator/iptables-alerter-xhfhc","openshift-ovn-kubernetes/ovnkube-node-lbj4t","kube-system/global-pull-secret-syncer-hrj5b","kube-system/konnectivity-agent-28km9","kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc","openshift-dns/node-resolver-mbkx7","openshift-image-registry/node-ca-f5cfv","openshift-multus/multus-4jvtj","openshift-multus/network-metrics-daemon-9b87x","openshift-cluster-node-tuning-operator/tuned-bkj7q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal","openshift-multus/multus-additional-cni-plugins-v8h6f"] Aug 04 00:02:35.888090 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.888072 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:35.889109 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.889086 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:35.889200 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:35.889154 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:02:35.890400 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.890244 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 00:02:35.890400 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.890262 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 00:02:35.890400 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.890278 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-shf2g\"" Aug 04 00:02:35.890400 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.890297 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 00:02:35.890598 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.890559 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.891590 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.891567 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:35.891691 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:35.891636 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj5b" podUID="51f02f93-efb0-42f3-acaf-2de4e4db0cbc" Aug 04 00:02:35.892628 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.892595 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 00:02:35.892738 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.892682 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lgxcd\"" Aug 04 00:02:35.893118 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.893100 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-28km9" Aug 04 00:02:35.893599 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.893583 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 00:02:35.893711 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.893631 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 00:02:35.893711 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.893691 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 00:02:35.893793 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.893748 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 00:02:35.893793 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.893764 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 00:02:35.894617 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.894598 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:35.895501 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.895465 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-plwr8\"" Aug 04 00:02:35.895839 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.895822 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f5cfv" Aug 04 00:02:35.896109 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.896088 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 00:02:35.896205 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.896174 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 00:02:35.896552 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.896537 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 00:02:35.896705 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.896688 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 00:02:35.896997 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.896827 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 00:02:35.896997 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.896856 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 00:02:35.896997 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.896905 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 00:02:35.897181 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.897106 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 00:02:35.897421 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.897393 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mbkx7" Aug 04 00:02:35.897421 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.897413 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2sdw4\"" Aug 04 00:02:35.897841 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.897823 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 00:02:35.897993 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.897975 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 00:02:35.898293 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.898275 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-76cpt\"" Aug 04 00:02:35.899268 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.899248 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 00:02:35.899354 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.899318 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xhfhc" Aug 04 00:02:35.900498 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.900468 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 00:02:35.900588 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.900564 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 00:02:35.902057 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.901074 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tqbpk\"" Aug 04 00:02:35.903031 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.903010 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 00:02:35.903206 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.903191 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 00:02:35.903312 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.903283 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-scmlx\"" Aug 04 00:02:35.903378 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.903358 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 00:02:35.903438 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.903388 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4jvtj" Aug 04 00:02:35.904149 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.904129 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:35.904238 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:35.904203 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqddv" podUID="8c2ca316-6027-4942-8085-22409a8b0b81" Aug 04 00:02:35.905124 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.905106 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:35.905343 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.905325 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 00:02:35.905640 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.905626 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zw9fr\"" Aug 04 00:02:35.906565 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.906548 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 00:02:35.907139 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907120 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vs7sl\"" Aug 04 00:02:35.907257 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907237 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs\") pod \"network-metrics-daemon-9b87x\" (UID: \"e656f284-112c-4309-9151-41e5a9f45ede\") " pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:35.907331 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907273 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-run-openvswitch\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.907331 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907298 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.907331 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907306 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 00:02:35.907331 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907323 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-kubelet-config\") pod \"global-pull-secret-syncer-hrj5b\" (UID: \"51f02f93-efb0-42f3-acaf-2de4e4db0cbc\") " pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:35.907527 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907347 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret\") pod \"global-pull-secret-syncer-hrj5b\" (UID: \"51f02f93-efb0-42f3-acaf-2de4e4db0cbc\") " pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:35.907527 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907390 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7866cfab-c563-4c2c-95d7-3485abec7465-system-cni-dir\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:35.907527 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907423 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7866cfab-c563-4c2c-95d7-3485abec7465-cni-binary-copy\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:35.907527 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907456 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-run-netns\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.907527 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907476 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-etc-openvswitch\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.907700 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907511 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-ovnkube-config\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.907700 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907556 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwtvm\" (UniqueName: \"kubernetes.io/projected/db57e300-ed1e-4192-9977-6fac6d117f66-kube-api-access-rwtvm\") pod \"node-ca-f5cfv\" (UID: \"db57e300-ed1e-4192-9977-6fac6d117f66\") " pod="openshift-image-registry/node-ca-f5cfv" Aug 04 00:02:35.907700 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907582 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:35.907700 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907604 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-run-systemd\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.907700 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907627 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-run-ovn\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.907700 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907658 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-node-log\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.907700 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907679 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-log-socket\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.908030 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907724 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-run-ovn-kubernetes\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.908030 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907762 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117-hosts-file\") pod \"node-resolver-mbkx7\" (UID: \"fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117\") " pod="openshift-dns/node-resolver-mbkx7" Aug 04 00:02:35.908030 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907787 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-registration-dir\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:35.908030 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907808 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7617213d-44e2-4227-bb9e-a9143683e61e-agent-certs\") pod \"konnectivity-agent-28km9\" (UID: \"7617213d-44e2-4227-bb9e-a9143683e61e\") " pod="kube-system/konnectivity-agent-28km9" Aug 04 00:02:35.908030 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907832 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 00:02:35.908030 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907853 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-ovn-node-metrics-cert\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.908030 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907879 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-dbus\") pod \"global-pull-secret-syncer-hrj5b\" (UID: \"51f02f93-efb0-42f3-acaf-2de4e4db0cbc\") " pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:35.908030 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907901 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7617213d-44e2-4227-bb9e-a9143683e61e-konnectivity-ca\") pod \"konnectivity-agent-28km9\" (UID: \"7617213d-44e2-4227-bb9e-a9143683e61e\") " pod="kube-system/konnectivity-agent-28km9" Aug 04 00:02:35.908030 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907925 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttxkb\" (UniqueName: \"kubernetes.io/projected/7866cfab-c563-4c2c-95d7-3485abec7465-kube-api-access-ttxkb\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:35.908030 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907948 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-slash\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.908030 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.907979 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-var-lib-openvswitch\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.908030 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908006 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-cni-netd\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.908030 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908031 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-socket-dir\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:35.908622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908067 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7866cfab-c563-4c2c-95d7-3485abec7465-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:35.908622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908101 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-kubelet\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.908622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908122 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-systemd-units\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.908622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908142 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-cni-bin\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.908622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908159 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkdrl\" (UniqueName: \"kubernetes.io/projected/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-kube-api-access-vkdrl\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.908622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908195 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117-tmp-dir\") pod \"node-resolver-mbkx7\" (UID: \"fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117\") " pod="openshift-dns/node-resolver-mbkx7" Aug 04 00:02:35.908622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908249 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcx6r\" (UniqueName: \"kubernetes.io/projected/fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117-kube-api-access-hcx6r\") pod \"node-resolver-mbkx7\" (UID: \"fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117\") " pod="openshift-dns/node-resolver-mbkx7" Aug 04 00:02:35.908622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908280 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp26h\" (UniqueName: \"kubernetes.io/projected/405d25c7-42d1-42bd-8bf0-06309e65436c-kube-api-access-zp26h\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:35.908622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908304 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7866cfab-c563-4c2c-95d7-3485abec7465-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:35.908622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908327 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db57e300-ed1e-4192-9977-6fac6d117f66-host\") pod \"node-ca-f5cfv\" (UID: \"db57e300-ed1e-4192-9977-6fac6d117f66\") " pod="openshift-image-registry/node-ca-f5cfv" Aug 04 00:02:35.908622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908348 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db57e300-ed1e-4192-9977-6fac6d117f66-serviceca\") pod \"node-ca-f5cfv\" (UID: \"db57e300-ed1e-4192-9977-6fac6d117f66\") " pod="openshift-image-registry/node-ca-f5cfv" Aug 04 00:02:35.908622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908370 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-env-overrides\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.908622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908391 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-ovnkube-script-lib\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:35.908622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908413 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-device-dir\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:35.908622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908435 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-etc-selinux\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:35.908622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908457 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7866cfab-c563-4c2c-95d7-3485abec7465-cnibin\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:35.909246 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908478 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8q2h\" (UniqueName: \"kubernetes.io/projected/e656f284-112c-4309-9151-41e5a9f45ede-kube-api-access-s8q2h\") pod \"network-metrics-daemon-9b87x\" (UID: \"e656f284-112c-4309-9151-41e5a9f45ede\") " pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:35.909246 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908499 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-sys-fs\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:35.909246 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908514 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7866cfab-c563-4c2c-95d7-3485abec7465-os-release\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:35.909246 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.908541 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7866cfab-c563-4c2c-95d7-3485abec7465-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:35.933847 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.933826 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hs7xn" Aug 04 00:02:35.939898 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.939878 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hs7xn" Aug 04 00:02:35.997500 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:35.997485 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 00:02:36.009017 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.008998 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-cni-bin\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.009128 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009023 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117-tmp-dir\") pod \"node-resolver-mbkx7\" (UID: \"fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117\") " pod="openshift-dns/node-resolver-mbkx7" Aug 04 00:02:36.009128 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009043 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcx6r\" (UniqueName: \"kubernetes.io/projected/fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117-kube-api-access-hcx6r\") pod \"node-resolver-mbkx7\" (UID: \"fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117\") " pod="openshift-dns/node-resolver-mbkx7" Aug 04 00:02:36.009128 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009065 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp26h\" (UniqueName: \"kubernetes.io/projected/405d25c7-42d1-42bd-8bf0-06309e65436c-kube-api-access-zp26h\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:36.009279 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009218 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-cni-bin\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.009279 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009269 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7866cfab-c563-4c2c-95d7-3485abec7465-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:36.009371 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009298 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-sysctl-conf\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.009371 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009328 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-env-overrides\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.009371 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009341 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117-tmp-dir\") pod \"node-resolver-mbkx7\" (UID: \"fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117\") " pod="openshift-dns/node-resolver-mbkx7" Aug 04 00:02:36.009371 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009353 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-etc-selinux\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:36.009563 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009395 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7866cfab-c563-4c2c-95d7-3485abec7465-cnibin\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:36.009563 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009436 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7866cfab-c563-4c2c-95d7-3485abec7465-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:36.009563 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009465 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-var-lib-kubelet\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.009563 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009500 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7866cfab-c563-4c2c-95d7-3485abec7465-cnibin\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:36.009563 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009490 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-multus-conf-dir\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.009563 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009545 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-host\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.009849 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009573 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-etc-selinux\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:36.009849 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009576 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs\") pod \"network-metrics-daemon-9b87x\" (UID: \"e656f284-112c-4309-9151-41e5a9f45ede\") " pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:36.009849 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009637 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-run-openvswitch\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.009849 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009668 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret\") pod \"global-pull-secret-syncer-hrj5b\" (UID: \"51f02f93-efb0-42f3-acaf-2de4e4db0cbc\") " pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:36.009849 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009697 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-run-netns\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.009849 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009725 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dbaf6a41-e197-4e72-bace-5b6621829d52-multus-daemon-config\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.009849 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009749 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpq74\" (UniqueName: \"kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74\") pod \"network-check-target-cqddv\" (UID: \"8c2ca316-6027-4942-8085-22409a8b0b81\") " pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:36.009849 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009773 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-run-netns\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.009849 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009795 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-ovnkube-config\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.009849 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:36.009802 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:36.009849 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009817 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:36.009849 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009843 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-multus-cni-dir\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.010406 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:36.009868 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs podName:e656f284-112c-4309-9151-41e5a9f45ede nodeName:}" failed. No retries permitted until 2026-08-04 00:02:36.509849147 +0000 UTC m=+2.992515476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs") pod "network-metrics-daemon-9b87x" (UID: "e656f284-112c-4309-9151-41e5a9f45ede") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:36.010406 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009894 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-run-openvswitch\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.010406 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009910 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-run-systemd\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.010406 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009929 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-env-overrides\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.010406 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009936 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-run-ovn\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.010406 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009980 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-node-log\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.010406 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:36.009992 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:36.010406 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010006 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-log-socket\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.010406 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010015 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-run-netns\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.010406 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010021 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7866cfab-c563-4c2c-95d7-3485abec7465-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:36.010406 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:36.010031 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret podName:51f02f93-efb0-42f3-acaf-2de4e4db0cbc nodeName:}" failed. No retries permitted until 2026-08-04 00:02:36.510020511 +0000 UTC m=+2.992686820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret") pod "global-pull-secret-syncer-hrj5b" (UID: "51f02f93-efb0-42f3-acaf-2de4e4db0cbc") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:36.010406 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.009988 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7866cfab-c563-4c2c-95d7-3485abec7465-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:36.010406 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010039 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-run-ovn\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.010406 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010045 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-run-ovn-kubernetes\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.010406 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010073 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:36.010406 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010080 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-run-systemd\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.010987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010076 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117-hosts-file\") pod \"node-resolver-mbkx7\" (UID: \"fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117\") " pod="openshift-dns/node-resolver-mbkx7" Aug 04 00:02:36.010987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010088 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-run-ovn-kubernetes\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.010987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010112 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117-hosts-file\") pod \"node-resolver-mbkx7\" (UID: \"fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117\") " pod="openshift-dns/node-resolver-mbkx7" Aug 04 00:02:36.010987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010116 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-multus-socket-dir-parent\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.010987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010118 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-log-socket\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.010987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010126 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-node-log\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.010987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010149 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-dbus\") pod \"global-pull-secret-syncer-hrj5b\" (UID: \"51f02f93-efb0-42f3-acaf-2de4e4db0cbc\") " pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:36.010987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010192 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7617213d-44e2-4227-bb9e-a9143683e61e-konnectivity-ca\") pod \"konnectivity-agent-28km9\" (UID: \"7617213d-44e2-4227-bb9e-a9143683e61e\") " pod="kube-system/konnectivity-agent-28km9" Aug 04 00:02:36.010987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010283 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttxkb\" (UniqueName: \"kubernetes.io/projected/7866cfab-c563-4c2c-95d7-3485abec7465-kube-api-access-ttxkb\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:36.010987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010333 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-dbus\") pod \"global-pull-secret-syncer-hrj5b\" (UID: \"51f02f93-efb0-42f3-acaf-2de4e4db0cbc\") " pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:36.010987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010370 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wrpv\" (UniqueName: \"kubernetes.io/projected/c1ed12d1-8a88-433e-a44e-69b44c34775e-kube-api-access-8wrpv\") pod \"iptables-alerter-xhfhc\" (UID: \"c1ed12d1-8a88-433e-a44e-69b44c34775e\") " pod="openshift-network-operator/iptables-alerter-xhfhc" Aug 04 00:02:36.010987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010411 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-system-cni-dir\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.010987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010436 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-cnibin\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.010987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010447 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-ovnkube-config\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.010987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010457 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbaf6a41-e197-4e72-bace-5b6621829d52-cni-binary-copy\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.010987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010481 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-run-multus-certs\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.010987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010507 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-slash\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.011471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010533 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-var-lib-openvswitch\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.011471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010553 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-cni-netd\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.011471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010590 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-slash\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.011471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010599 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-cni-netd\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.011471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010600 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-var-lib-openvswitch\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.011471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010622 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-socket-dir\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:36.011471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010652 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7866cfab-c563-4c2c-95d7-3485abec7465-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:36.011471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010693 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-systemd\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.011471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010715 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-run\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.011471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010733 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-socket-dir\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:36.011471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010768 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-kubelet\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.011471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010772 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7617213d-44e2-4227-bb9e-a9143683e61e-konnectivity-ca\") pod \"konnectivity-agent-28km9\" (UID: \"7617213d-44e2-4227-bb9e-a9143683e61e\") " pod="kube-system/konnectivity-agent-28km9" Aug 04 00:02:36.011471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010795 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-systemd-units\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.011471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010821 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkdrl\" (UniqueName: \"kubernetes.io/projected/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-kube-api-access-vkdrl\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.011471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010839 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-run-k8s-cni-cncf-io\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.011471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010847 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-kubelet\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.011471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010863 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-var-lib-cni-multus\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.012198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010865 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7866cfab-c563-4c2c-95d7-3485abec7465-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:36.012198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010878 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-sys\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.012198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010904 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-systemd-units\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.012198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010905 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-tuned\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.012198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010939 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db57e300-ed1e-4192-9977-6fac6d117f66-host\") pod \"node-ca-f5cfv\" (UID: \"db57e300-ed1e-4192-9977-6fac6d117f66\") " pod="openshift-image-registry/node-ca-f5cfv" Aug 04 00:02:36.012198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010954 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db57e300-ed1e-4192-9977-6fac6d117f66-serviceca\") pod \"node-ca-f5cfv\" (UID: \"db57e300-ed1e-4192-9977-6fac6d117f66\") " pod="openshift-image-registry/node-ca-f5cfv" Aug 04 00:02:36.012198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010968 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-ovnkube-script-lib\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.012198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010988 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db57e300-ed1e-4192-9977-6fac6d117f66-host\") pod \"node-ca-f5cfv\" (UID: \"db57e300-ed1e-4192-9977-6fac6d117f66\") " pod="openshift-image-registry/node-ca-f5cfv" Aug 04 00:02:36.012198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.010997 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-device-dir\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:36.012198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011020 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7866cfab-c563-4c2c-95d7-3485abec7465-os-release\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:36.012198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011044 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-kubernetes\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.012198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011068 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8q2h\" (UniqueName: \"kubernetes.io/projected/e656f284-112c-4309-9151-41e5a9f45ede-kube-api-access-s8q2h\") pod \"network-metrics-daemon-9b87x\" (UID: \"e656f284-112c-4309-9151-41e5a9f45ede\") " pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:36.012198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011093 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-sys-fs\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:36.012198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011117 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bda8b98a-34ac-4c55-97e6-03cda805a08f-tmp\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.012198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011142 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.012198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011151 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7866cfab-c563-4c2c-95d7-3485abec7465-os-release\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:36.012198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011094 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-device-dir\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:36.012955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011166 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-kubelet-config\") pod \"global-pull-secret-syncer-hrj5b\" (UID: \"51f02f93-efb0-42f3-acaf-2de4e4db0cbc\") " pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:36.012955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011201 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7866cfab-c563-4c2c-95d7-3485abec7465-system-cni-dir\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:36.012955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011210 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-kubelet-config\") pod \"global-pull-secret-syncer-hrj5b\" (UID: \"51f02f93-efb0-42f3-acaf-2de4e4db0cbc\") " pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:36.012955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011242 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7866cfab-c563-4c2c-95d7-3485abec7465-cni-binary-copy\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:36.012955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011270 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-hostroot\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.012955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011292 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-etc-kubernetes\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.012955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011296 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.012955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011313 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-sysconfig\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.012955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011334 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-etc-openvswitch\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.012955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011338 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7866cfab-c563-4c2c-95d7-3485abec7465-system-cni-dir\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:36.012955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011383 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db57e300-ed1e-4192-9977-6fac6d117f66-serviceca\") pod \"node-ca-f5cfv\" (UID: \"db57e300-ed1e-4192-9977-6fac6d117f66\") " pod="openshift-image-registry/node-ca-f5cfv" Aug 04 00:02:36.012955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011385 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-ovnkube-script-lib\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.012955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011412 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-sys-fs\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:36.012955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011448 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-etc-openvswitch\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.012955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011467 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwtvm\" (UniqueName: \"kubernetes.io/projected/db57e300-ed1e-4192-9977-6fac6d117f66-kube-api-access-rwtvm\") pod \"node-ca-f5cfv\" (UID: \"db57e300-ed1e-4192-9977-6fac6d117f66\") " pod="openshift-image-registry/node-ca-f5cfv" Aug 04 00:02:36.012955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011496 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ed12d1-8a88-433e-a44e-69b44c34775e-host-slash\") pod \"iptables-alerter-xhfhc\" (UID: \"c1ed12d1-8a88-433e-a44e-69b44c34775e\") " pod="openshift-network-operator/iptables-alerter-xhfhc" Aug 04 00:02:36.012955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011684 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-modprobe-d\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.013692 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011851 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1ed12d1-8a88-433e-a44e-69b44c34775e-iptables-alerter-script\") pod \"iptables-alerter-xhfhc\" (UID: \"c1ed12d1-8a88-433e-a44e-69b44c34775e\") " pod="openshift-network-operator/iptables-alerter-xhfhc" Aug 04 00:02:36.013692 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011893 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-registration-dir\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:36.013692 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011926 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7617213d-44e2-4227-bb9e-a9143683e61e-agent-certs\") pod \"konnectivity-agent-28km9\" (UID: \"7617213d-44e2-4227-bb9e-a9143683e61e\") " pod="kube-system/konnectivity-agent-28km9" Aug 04 00:02:36.013692 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011981 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/405d25c7-42d1-42bd-8bf0-06309e65436c-registration-dir\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:36.013692 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.011990 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-var-lib-cni-bin\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.013692 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.012033 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-lib-modules\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.013692 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.012040 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7866cfab-c563-4c2c-95d7-3485abec7465-cni-binary-copy\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:36.013692 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.012053 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzsq2\" (UniqueName: \"kubernetes.io/projected/bda8b98a-34ac-4c55-97e6-03cda805a08f-kube-api-access-xzsq2\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.013692 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.012075 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-ovn-node-metrics-cert\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.013692 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.012105 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv9tp\" (UniqueName: \"kubernetes.io/projected/dbaf6a41-e197-4e72-bace-5b6621829d52-kube-api-access-gv9tp\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.013692 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.012159 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-sysctl-d\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.013692 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.012194 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-var-lib-kubelet\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.013692 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.012253 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 00:02:36.013692 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.012262 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-os-release\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.016638 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.016616 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7617213d-44e2-4227-bb9e-a9143683e61e-agent-certs\") pod \"konnectivity-agent-28km9\" (UID: \"7617213d-44e2-4227-bb9e-a9143683e61e\") " pod="kube-system/konnectivity-agent-28km9" Aug 04 00:02:36.016978 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.016930 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-ovn-node-metrics-cert\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.020815 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.020490 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcx6r\" (UniqueName: \"kubernetes.io/projected/fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117-kube-api-access-hcx6r\") pod \"node-resolver-mbkx7\" (UID: \"fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117\") " pod="openshift-dns/node-resolver-mbkx7" Aug 04 00:02:36.020815 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.020774 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttxkb\" (UniqueName: \"kubernetes.io/projected/7866cfab-c563-4c2c-95d7-3485abec7465-kube-api-access-ttxkb\") pod \"multus-additional-cni-plugins-v8h6f\" (UID: \"7866cfab-c563-4c2c-95d7-3485abec7465\") " pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:36.021457 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.021439 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp26h\" (UniqueName: \"kubernetes.io/projected/405d25c7-42d1-42bd-8bf0-06309e65436c-kube-api-access-zp26h\") pod \"aws-ebs-csi-driver-node-nb4dc\" (UID: \"405d25c7-42d1-42bd-8bf0-06309e65436c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:36.022395 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.022107 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkdrl\" (UniqueName: \"kubernetes.io/projected/c7975ad9-b029-4d54-9d07-f4ff8bdd6d57-kube-api-access-vkdrl\") pod \"ovnkube-node-lbj4t\" (UID: \"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.022536 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.022516 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8q2h\" (UniqueName: \"kubernetes.io/projected/e656f284-112c-4309-9151-41e5a9f45ede-kube-api-access-s8q2h\") pod \"network-metrics-daemon-9b87x\" (UID: \"e656f284-112c-4309-9151-41e5a9f45ede\") " pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:36.023297 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.023266 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwtvm\" (UniqueName: \"kubernetes.io/projected/db57e300-ed1e-4192-9977-6fac6d117f66-kube-api-access-rwtvm\") pod \"node-ca-f5cfv\" (UID: \"db57e300-ed1e-4192-9977-6fac6d117f66\") " pod="openshift-image-registry/node-ca-f5cfv" Aug 04 00:02:36.113228 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113204 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-hostroot\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.113340 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113254 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-etc-kubernetes\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.113340 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113278 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-sysconfig\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.113340 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113302 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ed12d1-8a88-433e-a44e-69b44c34775e-host-slash\") pod \"iptables-alerter-xhfhc\" (UID: \"c1ed12d1-8a88-433e-a44e-69b44c34775e\") " pod="openshift-network-operator/iptables-alerter-xhfhc" Aug 04 00:02:36.113340 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113325 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-modprobe-d\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.113502 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113349 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1ed12d1-8a88-433e-a44e-69b44c34775e-iptables-alerter-script\") pod \"iptables-alerter-xhfhc\" (UID: \"c1ed12d1-8a88-433e-a44e-69b44c34775e\") " pod="openshift-network-operator/iptables-alerter-xhfhc" Aug 04 00:02:36.113502 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113373 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-var-lib-cni-bin\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.113502 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113395 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-lib-modules\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.113502 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113418 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzsq2\" (UniqueName: \"kubernetes.io/projected/bda8b98a-34ac-4c55-97e6-03cda805a08f-kube-api-access-xzsq2\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.113502 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113441 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv9tp\" (UniqueName: \"kubernetes.io/projected/dbaf6a41-e197-4e72-bace-5b6621829d52-kube-api-access-gv9tp\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.113502 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113463 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-sysctl-d\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.113502 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113486 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-var-lib-kubelet\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.113733 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113507 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-os-release\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.113733 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113536 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-sysctl-conf\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.113733 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113572 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-var-lib-kubelet\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.113733 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113592 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-multus-conf-dir\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.113733 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113608 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-host\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.113733 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113660 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-run-netns\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.113733 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113682 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dbaf6a41-e197-4e72-bace-5b6621829d52-multus-daemon-config\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.113733 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113709 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpq74\" (UniqueName: \"kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74\") pod \"network-check-target-cqddv\" (UID: \"8c2ca316-6027-4942-8085-22409a8b0b81\") " pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:36.114004 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113734 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-multus-cni-dir\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.114004 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113774 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-multus-socket-dir-parent\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.114004 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113799 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wrpv\" (UniqueName: \"kubernetes.io/projected/c1ed12d1-8a88-433e-a44e-69b44c34775e-kube-api-access-8wrpv\") pod \"iptables-alerter-xhfhc\" (UID: \"c1ed12d1-8a88-433e-a44e-69b44c34775e\") " pod="openshift-network-operator/iptables-alerter-xhfhc" Aug 04 00:02:36.114004 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113825 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-system-cni-dir\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.114004 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113846 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-cnibin\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.114004 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113868 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbaf6a41-e197-4e72-bace-5b6621829d52-cni-binary-copy\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.114004 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113891 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-run-multus-certs\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.114004 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113925 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-systemd\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.114004 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113947 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-run\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.114004 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.113985 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-run-k8s-cni-cncf-io\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.114410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114075 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-etc-kubernetes\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.114410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114085 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-systemd\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.114410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114125 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-multus-conf-dir\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.114410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114136 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-cnibin\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.114410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114175 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-var-lib-cni-bin\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.114410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114167 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-var-lib-kubelet\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.114410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114208 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-var-lib-cni-multus\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.114410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114252 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-sys\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.114410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114265 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-run\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.114410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114276 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-tuned\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.114410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114306 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-kubernetes\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.114410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114313 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-lib-modules\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.114410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114331 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bda8b98a-34ac-4c55-97e6-03cda805a08f-tmp\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.114410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114334 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-hostroot\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.114410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114347 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-run-netns\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.114410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114377 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-sysconfig\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.114410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114399 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-var-lib-cni-multus\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.115014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114438 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-sys\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.115014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114438 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-sysctl-d\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.115014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114480 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-sysctl-conf\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.115014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114497 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-modprobe-d\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.115014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114525 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-system-cni-dir\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.115014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114572 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dbaf6a41-e197-4e72-bace-5b6621829d52-multus-daemon-config\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.115014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114600 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbaf6a41-e197-4e72-bace-5b6621829d52-cni-binary-copy\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.115014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114616 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-var-lib-kubelet\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.115014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114307 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-run-k8s-cni-cncf-io\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.115014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114730 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-host\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.115014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114730 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-os-release\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.115014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114744 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-multus-cni-dir\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.115014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114770 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-multus-socket-dir-parent\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.115014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114791 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ed12d1-8a88-433e-a44e-69b44c34775e-host-slash\") pod \"iptables-alerter-xhfhc\" (UID: \"c1ed12d1-8a88-433e-a44e-69b44c34775e\") " pod="openshift-network-operator/iptables-alerter-xhfhc" Aug 04 00:02:36.115014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114804 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dbaf6a41-e197-4e72-bace-5b6621829d52-host-run-multus-certs\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.115014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.114848 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-kubernetes\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.115014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.115012 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1ed12d1-8a88-433e-a44e-69b44c34775e-iptables-alerter-script\") pod \"iptables-alerter-xhfhc\" (UID: \"c1ed12d1-8a88-433e-a44e-69b44c34775e\") " pod="openshift-network-operator/iptables-alerter-xhfhc" Aug 04 00:02:36.116552 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.116533 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bda8b98a-34ac-4c55-97e6-03cda805a08f-etc-tuned\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.116725 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.116707 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bda8b98a-34ac-4c55-97e6-03cda805a08f-tmp\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.120312 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:36.120017 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:02:36.120312 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:36.120039 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:02:36.120312 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:36.120051 2570 projected.go:196] Error preparing data for projected volume kube-api-access-mpq74 for pod openshift-network-diagnostics/network-check-target-cqddv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:36.120312 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:36.120131 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74 podName:8c2ca316-6027-4942-8085-22409a8b0b81 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:36.620094945 +0000 UTC m=+3.102761254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mpq74" (UniqueName: "kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74") pod "network-check-target-cqddv" (UID: "8c2ca316-6027-4942-8085-22409a8b0b81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:36.122049 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.122029 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzsq2\" (UniqueName: \"kubernetes.io/projected/bda8b98a-34ac-4c55-97e6-03cda805a08f-kube-api-access-xzsq2\") pod \"tuned-bkj7q\" (UID: \"bda8b98a-34ac-4c55-97e6-03cda805a08f\") " pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.122365 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.122346 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wrpv\" (UniqueName: \"kubernetes.io/projected/c1ed12d1-8a88-433e-a44e-69b44c34775e-kube-api-access-8wrpv\") pod \"iptables-alerter-xhfhc\" (UID: \"c1ed12d1-8a88-433e-a44e-69b44c34775e\") " pod="openshift-network-operator/iptables-alerter-xhfhc" Aug 04 00:02:36.122365 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.122359 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv9tp\" (UniqueName: \"kubernetes.io/projected/dbaf6a41-e197-4e72-bace-5b6621829d52-kube-api-access-gv9tp\") pod \"multus-4jvtj\" (UID: \"dbaf6a41-e197-4e72-bace-5b6621829d52\") " pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.151978 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.151934 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 00:02:36.206042 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.206021 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" Aug 04 00:02:36.213087 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.213068 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:02:36.221535 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.221508 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-28km9" Aug 04 00:02:36.226315 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.226298 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v8h6f" Aug 04 00:02:36.232782 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.232765 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f5cfv" Aug 04 00:02:36.240307 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.240287 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mbkx7" Aug 04 00:02:36.247755 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.247738 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xhfhc" Aug 04 00:02:36.254257 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.254241 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4jvtj" Aug 04 00:02:36.258917 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.258902 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" Aug 04 00:02:36.516490 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.516423 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs\") pod \"network-metrics-daemon-9b87x\" (UID: \"e656f284-112c-4309-9151-41e5a9f45ede\") " pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:36.516490 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.516459 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret\") pod \"global-pull-secret-syncer-hrj5b\" (UID: \"51f02f93-efb0-42f3-acaf-2de4e4db0cbc\") " pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:36.516664 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:36.516561 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:36.516664 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:36.516605 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:36.516664 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:36.516627 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs podName:e656f284-112c-4309-9151-41e5a9f45ede nodeName:}" failed. No retries permitted until 2026-08-04 00:02:37.516612919 +0000 UTC m=+3.999279226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs") pod "network-metrics-daemon-9b87x" (UID: "e656f284-112c-4309-9151-41e5a9f45ede") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:36.516664 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:36.516657 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret podName:51f02f93-efb0-42f3-acaf-2de4e4db0cbc nodeName:}" failed. No retries permitted until 2026-08-04 00:02:37.516640904 +0000 UTC m=+3.999307225 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret") pod "global-pull-secret-syncer-hrj5b" (UID: "51f02f93-efb0-42f3-acaf-2de4e4db0cbc") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:36.718882 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.718849 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpq74\" (UniqueName: \"kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74\") pod \"network-check-target-cqddv\" (UID: \"8c2ca316-6027-4942-8085-22409a8b0b81\") " pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:36.719044 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:36.719023 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:02:36.719122 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:36.719052 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:02:36.719122 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:36.719066 2570 projected.go:196] Error preparing data for projected volume kube-api-access-mpq74 for pod openshift-network-diagnostics/network-check-target-cqddv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:36.719214 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:36.719129 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74 podName:8c2ca316-6027-4942-8085-22409a8b0b81 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:37.719109821 +0000 UTC m=+4.201776150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpq74" (UniqueName: "kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74") pod "network-check-target-cqddv" (UID: "8c2ca316-6027-4942-8085-22409a8b0b81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:36.781356 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:36.781329 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7617213d_44e2_4227_bb9e_a9143683e61e.slice/crio-e928643b4b48cf633a2b1df898fd503e657512ba477b6909abbc051b6a490f80 WatchSource:0}: Error finding container e928643b4b48cf633a2b1df898fd503e657512ba477b6909abbc051b6a490f80: Status 404 returned error can't find the container with id e928643b4b48cf633a2b1df898fd503e657512ba477b6909abbc051b6a490f80 Aug 04 00:02:36.782699 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:36.782677 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405d25c7_42d1_42bd_8bf0_06309e65436c.slice/crio-5717cac050dd82d40b891f5f4395ad2d12a791ce9bac06d81b1b94fc68fde614 WatchSource:0}: Error finding container 5717cac050dd82d40b891f5f4395ad2d12a791ce9bac06d81b1b94fc68fde614: Status 404 returned error can't find the container with id 5717cac050dd82d40b891f5f4395ad2d12a791ce9bac06d81b1b94fc68fde614 Aug 04 00:02:36.785829 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:36.785803 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7866cfab_c563_4c2c_95d7_3485abec7465.slice/crio-3b7acba1213c9a17c4d98dba5d0267de26bd81a9ede42b56d9cebfe9f5de4df2 WatchSource:0}: Error finding container 3b7acba1213c9a17c4d98dba5d0267de26bd81a9ede42b56d9cebfe9f5de4df2: Status 404 returned error can't find the container with id 3b7acba1213c9a17c4d98dba5d0267de26bd81a9ede42b56d9cebfe9f5de4df2 Aug 04 00:02:36.786558 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:36.786488 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7975ad9_b029_4d54_9d07_f4ff8bdd6d57.slice/crio-fe1d37e22de3ed4ff9461e82ca2bfff11a39227b1c9ea49f7165e53572db4842 WatchSource:0}: Error finding container fe1d37e22de3ed4ff9461e82ca2bfff11a39227b1c9ea49f7165e53572db4842: Status 404 returned error can't find the container with id fe1d37e22de3ed4ff9461e82ca2bfff11a39227b1c9ea49f7165e53572db4842 Aug 04 00:02:36.787643 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:36.787627 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbaf6a41_e197_4e72_bace_5b6621829d52.slice/crio-aae904068c71eef56913621f854a215154a4655c40f53a69869d50960d733606 WatchSource:0}: Error finding container aae904068c71eef56913621f854a215154a4655c40f53a69869d50960d733606: Status 404 returned error can't find the container with id aae904068c71eef56913621f854a215154a4655c40f53a69869d50960d733606 Aug 04 00:02:36.789648 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:36.789455 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0ec9ec_6b13_46b6_9dc8_d1fdcfc8d117.slice/crio-3d7403783d967c205520fa6a06ba4081d0e1abf14cbfe22dcac46b4ea0bfa9db WatchSource:0}: Error finding container 3d7403783d967c205520fa6a06ba4081d0e1abf14cbfe22dcac46b4ea0bfa9db: Status 404 returned error can't find the container with id 3d7403783d967c205520fa6a06ba4081d0e1abf14cbfe22dcac46b4ea0bfa9db Aug 04 00:02:36.789836 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:36.789817 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda8b98a_34ac_4c55_97e6_03cda805a08f.slice/crio-bdd761681dc7a5095bb296749915101e3cb310d224c28050860d08b9a10218c6 WatchSource:0}: Error finding container bdd761681dc7a5095bb296749915101e3cb310d224c28050860d08b9a10218c6: Status 404 returned error can't find the container with id bdd761681dc7a5095bb296749915101e3cb310d224c28050860d08b9a10218c6 Aug 04 00:02:36.791819 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:36.791788 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb57e300_ed1e_4192_9977_6fac6d117f66.slice/crio-a1bc778cd95ef39bc94d8d3ca0c98660df53e29262aa77d6abb2f47a12af4d8c WatchSource:0}: Error finding container a1bc778cd95ef39bc94d8d3ca0c98660df53e29262aa77d6abb2f47a12af4d8c: Status 404 returned error can't find the container with id a1bc778cd95ef39bc94d8d3ca0c98660df53e29262aa77d6abb2f47a12af4d8c Aug 04 00:02:36.792555 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:02:36.792532 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ed12d1_8a88_433e_a44e_69b44c34775e.slice/crio-cbc41b57aec6b8b5ebda4f7abb89c40c1bde9ff46555e7e4e08561415d33cc00 WatchSource:0}: Error finding container cbc41b57aec6b8b5ebda4f7abb89c40c1bde9ff46555e7e4e08561415d33cc00: Status 404 returned error can't find the container with id cbc41b57aec6b8b5ebda4f7abb89c40c1bde9ff46555e7e4e08561415d33cc00 Aug 04 00:02:36.942302 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.942260 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-02 23:57:35 +0000 UTC" deadline="2028-04-05 00:45:07.73871003 +0000 UTC" Aug 04 00:02:36.942302 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:36.942296 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14640h42m30.796416615s" Aug 04 00:02:37.036069 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:37.036009 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal" event={"ID":"da457b2ccc5cb1e2a6488c095c0a0489","Type":"ContainerStarted","Data":"2812dbbeb2195895c21a1ae33d67189c377345cc8250c46e3d64440e4bddda07"} Aug 04 00:02:37.037101 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:37.037077 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xhfhc" event={"ID":"c1ed12d1-8a88-433e-a44e-69b44c34775e","Type":"ContainerStarted","Data":"cbc41b57aec6b8b5ebda4f7abb89c40c1bde9ff46555e7e4e08561415d33cc00"} Aug 04 00:02:37.038050 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:37.038028 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f5cfv" event={"ID":"db57e300-ed1e-4192-9977-6fac6d117f66","Type":"ContainerStarted","Data":"a1bc778cd95ef39bc94d8d3ca0c98660df53e29262aa77d6abb2f47a12af4d8c"} Aug 04 00:02:37.039030 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:37.039004 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mbkx7" event={"ID":"fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117","Type":"ContainerStarted","Data":"3d7403783d967c205520fa6a06ba4081d0e1abf14cbfe22dcac46b4ea0bfa9db"} Aug 04 00:02:37.040022 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:37.040002 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8h6f" event={"ID":"7866cfab-c563-4c2c-95d7-3485abec7465","Type":"ContainerStarted","Data":"3b7acba1213c9a17c4d98dba5d0267de26bd81a9ede42b56d9cebfe9f5de4df2"} Aug 04 00:02:37.040937 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:37.040919 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" event={"ID":"405d25c7-42d1-42bd-8bf0-06309e65436c","Type":"ContainerStarted","Data":"5717cac050dd82d40b891f5f4395ad2d12a791ce9bac06d81b1b94fc68fde614"} Aug 04 00:02:37.041840 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:37.041821 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-28km9" event={"ID":"7617213d-44e2-4227-bb9e-a9143683e61e","Type":"ContainerStarted","Data":"e928643b4b48cf633a2b1df898fd503e657512ba477b6909abbc051b6a490f80"} Aug 04 00:02:37.042731 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:37.042712 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" event={"ID":"bda8b98a-34ac-4c55-97e6-03cda805a08f","Type":"ContainerStarted","Data":"bdd761681dc7a5095bb296749915101e3cb310d224c28050860d08b9a10218c6"} Aug 04 00:02:37.043649 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:37.043631 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4jvtj" event={"ID":"dbaf6a41-e197-4e72-bace-5b6621829d52","Type":"ContainerStarted","Data":"aae904068c71eef56913621f854a215154a4655c40f53a69869d50960d733606"} Aug 04 00:02:37.044565 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:37.044547 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" event={"ID":"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57","Type":"ContainerStarted","Data":"fe1d37e22de3ed4ff9461e82ca2bfff11a39227b1c9ea49f7165e53572db4842"} Aug 04 00:02:37.049771 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:37.049739 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-102.ec2.internal" podStartSLOduration=2.049724581 podStartE2EDuration="2.049724581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:02:37.049696173 +0000 UTC m=+3.532362503" watchObservedRunningTime="2026-08-04 00:02:37.049724581 +0000 UTC m=+3.532390913" Aug 04 00:02:37.525327 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:37.525162 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs\") pod \"network-metrics-daemon-9b87x\" (UID: \"e656f284-112c-4309-9151-41e5a9f45ede\") " pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:37.525327 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:37.525210 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret\") pod \"global-pull-secret-syncer-hrj5b\" (UID: \"51f02f93-efb0-42f3-acaf-2de4e4db0cbc\") " pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:37.525583 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:37.525395 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:37.525583 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:37.525470 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret podName:51f02f93-efb0-42f3-acaf-2de4e4db0cbc nodeName:}" failed. No retries permitted until 2026-08-04 00:02:39.525450605 +0000 UTC m=+6.008116917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret") pod "global-pull-secret-syncer-hrj5b" (UID: "51f02f93-efb0-42f3-acaf-2de4e4db0cbc") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:37.525962 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:37.525855 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:37.525962 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:37.525930 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs podName:e656f284-112c-4309-9151-41e5a9f45ede nodeName:}" failed. No retries permitted until 2026-08-04 00:02:39.525912009 +0000 UTC m=+6.008578319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs") pod "network-metrics-daemon-9b87x" (UID: "e656f284-112c-4309-9151-41e5a9f45ede") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:37.726613 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:37.726483 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpq74\" (UniqueName: \"kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74\") pod \"network-check-target-cqddv\" (UID: \"8c2ca316-6027-4942-8085-22409a8b0b81\") " pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:37.726783 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:37.726631 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:02:37.726783 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:37.726652 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:02:37.726783 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:37.726664 2570 projected.go:196] Error preparing data for projected volume kube-api-access-mpq74 for pod openshift-network-diagnostics/network-check-target-cqddv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:37.726783 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:37.726721 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74 podName:8c2ca316-6027-4942-8085-22409a8b0b81 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:39.726702677 +0000 UTC m=+6.209369006 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpq74" (UniqueName: "kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74") pod "network-check-target-cqddv" (UID: "8c2ca316-6027-4942-8085-22409a8b0b81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:37.943554 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:37.943296 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-02 23:57:35 +0000 UTC" deadline="2028-01-13 13:49:42.084109044 +0000 UTC" Aug 04 00:02:37.943554 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:37.943550 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12661h47m4.140565686s" Aug 04 00:02:38.029659 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:38.028360 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:38.029659 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:38.028495 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:02:38.029659 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:38.028868 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:38.029659 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:38.028957 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj5b" podUID="51f02f93-efb0-42f3-acaf-2de4e4db0cbc" Aug 04 00:02:38.029659 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:38.029056 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:38.029659 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:38.029132 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqddv" podUID="8c2ca316-6027-4942-8085-22409a8b0b81" Aug 04 00:02:39.543201 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:39.543167 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs\") pod \"network-metrics-daemon-9b87x\" (UID: \"e656f284-112c-4309-9151-41e5a9f45ede\") " pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:39.543680 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:39.543215 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret\") pod \"global-pull-secret-syncer-hrj5b\" (UID: \"51f02f93-efb0-42f3-acaf-2de4e4db0cbc\") " pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:39.543680 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:39.543387 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:39.543680 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:39.543447 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret podName:51f02f93-efb0-42f3-acaf-2de4e4db0cbc nodeName:}" failed. No retries permitted until 2026-08-04 00:02:43.543424109 +0000 UTC m=+10.026090420 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret") pod "global-pull-secret-syncer-hrj5b" (UID: "51f02f93-efb0-42f3-acaf-2de4e4db0cbc") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:39.543858 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:39.543816 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:39.543910 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:39.543862 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs podName:e656f284-112c-4309-9151-41e5a9f45ede nodeName:}" failed. No retries permitted until 2026-08-04 00:02:43.543847618 +0000 UTC m=+10.026513932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs") pod "network-metrics-daemon-9b87x" (UID: "e656f284-112c-4309-9151-41e5a9f45ede") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:39.745322 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:39.744820 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpq74\" (UniqueName: \"kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74\") pod \"network-check-target-cqddv\" (UID: \"8c2ca316-6027-4942-8085-22409a8b0b81\") " pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:39.745322 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:39.744993 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:02:39.745322 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:39.745011 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:02:39.745322 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:39.745024 2570 projected.go:196] Error preparing data for projected volume kube-api-access-mpq74 for pod openshift-network-diagnostics/network-check-target-cqddv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:39.745322 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:39.745075 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74 podName:8c2ca316-6027-4942-8085-22409a8b0b81 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:43.745058964 +0000 UTC m=+10.227725276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpq74" (UniqueName: "kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74") pod "network-check-target-cqddv" (UID: "8c2ca316-6027-4942-8085-22409a8b0b81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:40.027594 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:40.027517 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:40.027740 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:40.027662 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:02:40.027942 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:40.027911 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:40.028166 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:40.028033 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:40.028166 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:40.028034 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj5b" podUID="51f02f93-efb0-42f3-acaf-2de4e4db0cbc" Aug 04 00:02:40.028166 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:40.028134 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqddv" podUID="8c2ca316-6027-4942-8085-22409a8b0b81" Aug 04 00:02:42.028034 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:42.028003 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:42.028461 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:42.028003 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:42.028461 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:42.028131 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqddv" podUID="8c2ca316-6027-4942-8085-22409a8b0b81" Aug 04 00:02:42.028461 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:42.028178 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:42.028461 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:42.028299 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:02:42.028461 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:42.028398 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj5b" podUID="51f02f93-efb0-42f3-acaf-2de4e4db0cbc" Aug 04 00:02:43.573197 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:43.573163 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs\") pod \"network-metrics-daemon-9b87x\" (UID: \"e656f284-112c-4309-9151-41e5a9f45ede\") " pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:43.573635 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:43.573204 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret\") pod \"global-pull-secret-syncer-hrj5b\" (UID: \"51f02f93-efb0-42f3-acaf-2de4e4db0cbc\") " pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:43.573635 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:43.573342 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:43.573635 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:43.573412 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs podName:e656f284-112c-4309-9151-41e5a9f45ede nodeName:}" failed. No retries permitted until 2026-08-04 00:02:51.573392152 +0000 UTC m=+18.056058474 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs") pod "network-metrics-daemon-9b87x" (UID: "e656f284-112c-4309-9151-41e5a9f45ede") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:43.573635 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:43.573349 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:43.573635 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:43.573504 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret podName:51f02f93-efb0-42f3-acaf-2de4e4db0cbc nodeName:}" failed. No retries permitted until 2026-08-04 00:02:51.573490404 +0000 UTC m=+18.056156711 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret") pod "global-pull-secret-syncer-hrj5b" (UID: "51f02f93-efb0-42f3-acaf-2de4e4db0cbc") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:43.774662 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:43.774623 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpq74\" (UniqueName: \"kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74\") pod \"network-check-target-cqddv\" (UID: \"8c2ca316-6027-4942-8085-22409a8b0b81\") " pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:43.774850 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:43.774774 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:02:43.774850 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:43.774800 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:02:43.774850 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:43.774813 2570 projected.go:196] Error preparing data for projected volume kube-api-access-mpq74 for pod openshift-network-diagnostics/network-check-target-cqddv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:43.775020 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:43.774872 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74 podName:8c2ca316-6027-4942-8085-22409a8b0b81 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:51.774853972 +0000 UTC m=+18.257520295 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpq74" (UniqueName: "kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74") pod "network-check-target-cqddv" (UID: "8c2ca316-6027-4942-8085-22409a8b0b81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:44.029136 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:44.028397 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:44.029136 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:44.028500 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj5b" podUID="51f02f93-efb0-42f3-acaf-2de4e4db0cbc" Aug 04 00:02:44.029136 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:44.028835 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:44.029136 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:44.028922 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqddv" podUID="8c2ca316-6027-4942-8085-22409a8b0b81" Aug 04 00:02:44.029136 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:44.028971 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:44.029136 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:44.029058 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:02:46.028177 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:46.028106 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:46.028667 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:46.028298 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqddv" podUID="8c2ca316-6027-4942-8085-22409a8b0b81" Aug 04 00:02:46.028667 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:46.028115 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:46.028797 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:46.028726 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:46.028852 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:46.028828 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:02:46.028966 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:46.028945 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj5b" podUID="51f02f93-efb0-42f3-acaf-2de4e4db0cbc" Aug 04 00:02:48.027460 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:48.027372 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:48.027460 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:48.027411 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:48.027940 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:48.027376 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:48.027940 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:48.027509 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj5b" podUID="51f02f93-efb0-42f3-acaf-2de4e4db0cbc" Aug 04 00:02:48.027940 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:48.027561 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqddv" podUID="8c2ca316-6027-4942-8085-22409a8b0b81" Aug 04 00:02:48.027940 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:48.027652 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:02:50.027580 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:50.027548 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:50.028019 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:50.027548 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:50.028019 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:50.027663 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqddv" podUID="8c2ca316-6027-4942-8085-22409a8b0b81" Aug 04 00:02:50.028019 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:50.027749 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj5b" podUID="51f02f93-efb0-42f3-acaf-2de4e4db0cbc" Aug 04 00:02:50.028019 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:50.027554 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:50.028019 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:50.027848 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:02:51.634376 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:51.634351 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs\") pod \"network-metrics-daemon-9b87x\" (UID: \"e656f284-112c-4309-9151-41e5a9f45ede\") " pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:51.634769 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:51.634383 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret\") pod \"global-pull-secret-syncer-hrj5b\" (UID: \"51f02f93-efb0-42f3-acaf-2de4e4db0cbc\") " pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:51.634769 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:51.634519 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:51.634769 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:51.634543 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:51.634769 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:51.634586 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs podName:e656f284-112c-4309-9151-41e5a9f45ede nodeName:}" failed. No retries permitted until 2026-08-04 00:03:07.634565289 +0000 UTC m=+34.117231611 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs") pod "network-metrics-daemon-9b87x" (UID: "e656f284-112c-4309-9151-41e5a9f45ede") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:51.634769 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:51.634607 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret podName:51f02f93-efb0-42f3-acaf-2de4e4db0cbc nodeName:}" failed. No retries permitted until 2026-08-04 00:03:07.634596232 +0000 UTC m=+34.117262548 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret") pod "global-pull-secret-syncer-hrj5b" (UID: "51f02f93-efb0-42f3-acaf-2de4e4db0cbc") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:51.835741 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:51.835709 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpq74\" (UniqueName: \"kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74\") pod \"network-check-target-cqddv\" (UID: \"8c2ca316-6027-4942-8085-22409a8b0b81\") " pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:51.835904 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:51.835849 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:02:51.835904 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:51.835869 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:02:51.835904 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:51.835889 2570 projected.go:196] Error preparing data for projected volume kube-api-access-mpq74 for pod openshift-network-diagnostics/network-check-target-cqddv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:51.836016 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:51.835936 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74 podName:8c2ca316-6027-4942-8085-22409a8b0b81 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:07.835924426 +0000 UTC m=+34.318590734 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpq74" (UniqueName: "kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74") pod "network-check-target-cqddv" (UID: "8c2ca316-6027-4942-8085-22409a8b0b81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:52.027378 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:52.027305 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:52.027533 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:52.027312 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:52.027533 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:52.027418 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqddv" podUID="8c2ca316-6027-4942-8085-22409a8b0b81" Aug 04 00:02:52.027654 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:52.027529 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:02:52.027654 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:52.027317 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:52.027654 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:52.027630 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj5b" podUID="51f02f93-efb0-42f3-acaf-2de4e4db0cbc" Aug 04 00:02:54.028319 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:54.028276 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:54.028767 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:54.028376 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:54.028767 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:54.028386 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj5b" podUID="51f02f93-efb0-42f3-acaf-2de4e4db0cbc" Aug 04 00:02:54.028767 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:54.028446 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqddv" podUID="8c2ca316-6027-4942-8085-22409a8b0b81" Aug 04 00:02:54.028767 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:54.028494 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:54.028767 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:54.028566 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:02:56.028462 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:56.028356 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:56.028462 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:56.028401 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:56.028907 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:56.028356 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:56.028907 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:56.028503 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:02:56.028907 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:56.028604 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqddv" podUID="8c2ca316-6027-4942-8085-22409a8b0b81" Aug 04 00:02:56.028907 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:56.028688 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj5b" podUID="51f02f93-efb0-42f3-acaf-2de4e4db0cbc" Aug 04 00:02:56.099788 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:56.099614 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" event={"ID":"bda8b98a-34ac-4c55-97e6-03cda805a08f","Type":"ContainerStarted","Data":"ad1bb9c3336075907e97cd87f8380a933022d7aa718c9fb0434db2e39923c4eb"} Aug 04 00:02:56.116734 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:56.116687 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bkj7q" podStartSLOduration=3.201743382 podStartE2EDuration="22.116668054s" totalImagesPullingTime="18.914924672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:02:56.115994394 +0000 UTC m=+22.598660801" watchObservedRunningTime="2026-08-04 00:02:56.116668054 +0000 UTC m=+22.599334385" Aug 04 00:02:56.120109 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:56.120085 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:02:56.120528 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:56.120498 2570 generic.go:356] "Generic (PLEG): container finished" podID="c7975ad9-b029-4d54-9d07-f4ff8bdd6d57" containerID="cc2fbd24f8299b6c2a327e63de5238b5d5792bc481340ef64a5c325fb8f24269" exitCode=1 Aug 04 00:02:56.120602 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:56.120550 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" event={"ID":"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57","Type":"ContainerStarted","Data":"322d402c8ff6e46646c8ba4caebebec397dcf02e51ea598577af5d419d895e6a"} Aug 04 00:02:56.120671 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:56.120623 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" event={"ID":"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57","Type":"ContainerDied","Data":"cc2fbd24f8299b6c2a327e63de5238b5d5792bc481340ef64a5c325fb8f24269"} Aug 04 00:02:56.120729 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:56.120677 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" event={"ID":"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57","Type":"ContainerStarted","Data":"5faa28e0dec8aa43bfe018529befba92b83b7400a466efcb71a1ad384ca2f145"} Aug 04 00:02:57.124762 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.124554 2570 generic.go:356] "Generic (PLEG): container finished" podID="470b6ba76b1c422c0a4df3ff57e9cabb" containerID="229e9df2751df07281efffcd6f673b329ab0cb64de76beeb34012a8a7d6743f5" exitCode=0 Aug 04 00:02:57.125357 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.124627 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal" event={"ID":"470b6ba76b1c422c0a4df3ff57e9cabb","Type":"ContainerDied","Data":"229e9df2751df07281efffcd6f673b329ab0cb64de76beeb34012a8a7d6743f5"} Aug 04 00:02:57.127047 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.127015 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xhfhc" event={"ID":"c1ed12d1-8a88-433e-a44e-69b44c34775e","Type":"ContainerStarted","Data":"17830371b81b0c333876ce8dc2c739d1debd6755c28ca34d6d9ba50361619b6a"} Aug 04 00:02:57.131745 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.131713 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f5cfv" event={"ID":"db57e300-ed1e-4192-9977-6fac6d117f66","Type":"ContainerStarted","Data":"570275136d30fc7d2e38e49815c0ecd651ac5116774ba6e242b43dd8c6b3f911"} Aug 04 00:02:57.133359 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.133320 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mbkx7" event={"ID":"fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117","Type":"ContainerStarted","Data":"10f8540f77c5a973242444af331d003367921b68b83a28bd3c24223fe7611f68"} Aug 04 00:02:57.134980 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.134958 2570 generic.go:356] "Generic (PLEG): container finished" podID="7866cfab-c563-4c2c-95d7-3485abec7465" containerID="f0f749107ec46346a4604faa019b9f7a919f3dc8725361c6473ad3ea5c21db1c" exitCode=0 Aug 04 00:02:57.135081 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.135034 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8h6f" event={"ID":"7866cfab-c563-4c2c-95d7-3485abec7465","Type":"ContainerDied","Data":"f0f749107ec46346a4604faa019b9f7a919f3dc8725361c6473ad3ea5c21db1c"} Aug 04 00:02:57.136803 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.136780 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" event={"ID":"405d25c7-42d1-42bd-8bf0-06309e65436c","Type":"ContainerStarted","Data":"e66b347b6b316c724e43822993e9ba089996348ed66ef87440774e1972f33562"} Aug 04 00:02:57.138576 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.138553 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-28km9" event={"ID":"7617213d-44e2-4227-bb9e-a9143683e61e","Type":"ContainerStarted","Data":"34488022a767d816eb129e085cc973d1642ebc9fc9142a92f25f565f8c8b44df"} Aug 04 00:02:57.140276 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.140251 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4jvtj" event={"ID":"dbaf6a41-e197-4e72-bace-5b6621829d52","Type":"ContainerStarted","Data":"05e347d66a51db71c3fcfb7477157371b2ac3d41ba40179208b30b74f01aa9c0"} Aug 04 00:02:57.145459 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.145443 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:02:57.146087 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.146055 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" event={"ID":"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57","Type":"ContainerStarted","Data":"66c533d7ea25a06fc6fd1d87c70349c1ffa5c9d9594d0cb275955de87fe1b4da"} Aug 04 00:02:57.146183 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.146126 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" event={"ID":"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57","Type":"ContainerStarted","Data":"161dfcca10e2a2e715aa62117b9114703794b8cc6dd7eac4b09c385aa77ede03"} Aug 04 00:02:57.146183 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.146176 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" event={"ID":"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57","Type":"ContainerStarted","Data":"517f13b9fde5fbece67fba928949b3819bdd0b9b55b8ff53163cd025ac582d5b"} Aug 04 00:02:57.183309 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.183273 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f5cfv" podStartSLOduration=4.272031473 podStartE2EDuration="23.18326168s" totalImagesPullingTime="18.911230207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:02:57.183200994 +0000 UTC m=+23.665867324" watchObservedRunningTime="2026-08-04 00:02:57.18326168 +0000 UTC m=+23.665928072" Aug 04 00:02:57.198756 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.198724 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mbkx7" podStartSLOduration=12.309069296 podStartE2EDuration="23.198713609s" totalImagesPullingTime="10.889644313s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:02:57.198537957 +0000 UTC m=+23.681204292" watchObservedRunningTime="2026-08-04 00:02:57.198713609 +0000 UTC m=+23.681379938" Aug 04 00:02:57.233378 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.233276 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-28km9" podStartSLOduration=8.394414092 podStartE2EDuration="23.233260831s" totalImagesPullingTime="14.838846739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:02:57.218796837 +0000 UTC m=+23.701463166" watchObservedRunningTime="2026-08-04 00:02:57.233260831 +0000 UTC m=+23.715927163" Aug 04 00:02:57.253749 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.253719 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xhfhc" podStartSLOduration=4.343103375 podStartE2EDuration="23.253707624s" totalImagesPullingTime="18.910604249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:02:57.233492791 +0000 UTC m=+23.716159122" watchObservedRunningTime="2026-08-04 00:02:57.253707624 +0000 UTC m=+23.736373971" Aug 04 00:02:57.254031 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.254012 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4jvtj" podStartSLOduration=4.015864803 podStartE2EDuration="23.254006773s" totalImagesPullingTime="19.23814197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:02:57.253569272 +0000 UTC m=+23.736235614" watchObservedRunningTime="2026-08-04 00:02:57.254006773 +0000 UTC m=+23.736673102" Aug 04 00:02:57.529133 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.529108 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 00:02:57.974887 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.974795 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T00:02:57.529129662Z","UUID":"eae64a1c-7f12-40bd-8e60-b18fbef1584f","Handler":null,"Name":"","Endpoint":""} Aug 04 00:02:57.976404 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.976386 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 00:02:57.976543 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:57.976422 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 00:02:58.027473 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:58.027451 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:02:58.027636 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:58.027451 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:02:58.027636 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:58.027560 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj5b" podUID="51f02f93-efb0-42f3-acaf-2de4e4db0cbc" Aug 04 00:02:58.027753 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:58.027657 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:02:58.027753 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:58.027451 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:02:58.027753 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:02:58.027743 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqddv" podUID="8c2ca316-6027-4942-8085-22409a8b0b81" Aug 04 00:02:58.150430 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:58.150402 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal" event={"ID":"470b6ba76b1c422c0a4df3ff57e9cabb","Type":"ContainerStarted","Data":"5bc9514abfc10fbdbf01dfb9846a52ebb5c7d7fca12441dc553ec2bdd5514894"} Aug 04 00:02:58.153435 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:58.152943 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" event={"ID":"405d25c7-42d1-42bd-8bf0-06309e65436c","Type":"ContainerStarted","Data":"f61d2dde68336048c65d170dd19d44b7f5168c9f70f5b2d74e1872f281ffe422"} Aug 04 00:02:58.168394 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:58.168342 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-102.ec2.internal" podStartSLOduration=23.168324928 podStartE2EDuration="23.168324928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:02:58.166828837 +0000 UTC m=+24.649495170" watchObservedRunningTime="2026-08-04 00:02:58.168324928 +0000 UTC m=+24.650991258" Aug 04 00:02:59.157372 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:59.157323 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" event={"ID":"405d25c7-42d1-42bd-8bf0-06309e65436c","Type":"ContainerStarted","Data":"279b61d8d6383d89050e25acf67c93773657112963bd5bce0bdb7e1165ef5627"} Aug 04 00:02:59.160821 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:59.160797 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:02:59.161292 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:59.161263 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" event={"ID":"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57","Type":"ContainerStarted","Data":"4362c8b98c4388e392b965f0e14e3f70d1d83594e207f479539702c9e7f3a5d6"} Aug 04 00:02:59.174442 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:59.174400 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nb4dc" podStartSLOduration=3.730523743 podStartE2EDuration="25.174387633s" totalImagesPullingTime="21.44386389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:02:59.17408231 +0000 UTC m=+25.656748642" watchObservedRunningTime="2026-08-04 00:02:59.174387633 +0000 UTC m=+25.657053964" Aug 04 00:02:59.565796 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:59.565763 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-28km9" Aug 04 00:02:59.566458 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:02:59.566438 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-28km9" Aug 04 00:03:00.027921 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:00.027889 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:03:00.027921 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:00.027918 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:03:00.028146 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:00.027890 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:03:00.028146 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:00.028019 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqddv" podUID="8c2ca316-6027-4942-8085-22409a8b0b81" Aug 04 00:03:00.028146 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:00.028101 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj5b" podUID="51f02f93-efb0-42f3-acaf-2de4e4db0cbc" Aug 04 00:03:00.028328 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:00.028163 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:03:00.164164 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:00.164137 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-28km9" Aug 04 00:03:00.164676 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:00.164660 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-28km9" Aug 04 00:03:01.174768 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:01.173337 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:03:01.175248 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:01.174803 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" event={"ID":"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57","Type":"ContainerStarted","Data":"0c6915af086a5b3250ea13b701717dcb68d22f67a14f89952e13330714ff93ea"} Aug 04 00:03:01.175590 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:01.175278 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:03:01.176077 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:01.175924 2570 scope.go:117] "RemoveContainer" containerID="cc2fbd24f8299b6c2a327e63de5238b5d5792bc481340ef64a5c325fb8f24269" Aug 04 00:03:01.194736 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:01.194627 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:03:02.028081 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:02.028050 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:03:02.028249 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:02.028149 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:03:02.028249 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:02.028170 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj5b" podUID="51f02f93-efb0-42f3-acaf-2de4e4db0cbc" Aug 04 00:03:02.028249 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:02.028154 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:03:02.028423 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:02.028287 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqddv" podUID="8c2ca316-6027-4942-8085-22409a8b0b81" Aug 04 00:03:02.028423 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:02.028319 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:03:02.178078 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:02.178015 2570 generic.go:356] "Generic (PLEG): container finished" podID="7866cfab-c563-4c2c-95d7-3485abec7465" containerID="84ab61c102e9d7f006fc5b8b2b720df06e6e42c4d7e325561521fe0caf5646ea" exitCode=0 Aug 04 00:03:02.178839 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:02.178104 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8h6f" event={"ID":"7866cfab-c563-4c2c-95d7-3485abec7465","Type":"ContainerDied","Data":"84ab61c102e9d7f006fc5b8b2b720df06e6e42c4d7e325561521fe0caf5646ea"} Aug 04 00:03:02.182692 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:02.182666 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:03:02.183130 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:02.183089 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" event={"ID":"c7975ad9-b029-4d54-9d07-f4ff8bdd6d57","Type":"ContainerStarted","Data":"0e7c4ce2774a60db7fdb901e0e832e18225613d5bb3a51184e70fc4c9096db16"} Aug 04 00:03:02.183256 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:02.183217 2570 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Aug 04 00:03:02.183422 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:02.183403 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:03:02.199904 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:02.199879 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:03:02.225271 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:02.225211 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" podStartSLOduration=9.285598652000001 podStartE2EDuration="28.225198993s" totalImagesPullingTime="18.939600341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:02.224824364 +0000 UTC m=+28.707490686" watchObservedRunningTime="2026-08-04 00:03:02.225198993 +0000 UTC m=+28.707865322" Aug 04 00:03:03.039280 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:03.039055 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:03:03.104040 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:03.104018 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrj5b"] Aug 04 00:03:03.104125 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:03.104118 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:03:03.104208 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:03.104190 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj5b" podUID="51f02f93-efb0-42f3-acaf-2de4e4db0cbc" Aug 04 00:03:03.106694 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:03.106661 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cqddv"] Aug 04 00:03:03.106811 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:03.106775 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:03:03.106900 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:03.106872 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqddv" podUID="8c2ca316-6027-4942-8085-22409a8b0b81" Aug 04 00:03:03.107382 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:03.107362 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9b87x"] Aug 04 00:03:03.107483 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:03.107466 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:03:03.107570 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:03.107552 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:03:03.187340 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:03.187314 2570 generic.go:356] "Generic (PLEG): container finished" podID="7866cfab-c563-4c2c-95d7-3485abec7465" containerID="361e9b935edede4b958765527a76c4781f36ada19e305b6e2c687ceec5ed9c44" exitCode=0 Aug 04 00:03:03.187648 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:03.187400 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8h6f" event={"ID":"7866cfab-c563-4c2c-95d7-3485abec7465","Type":"ContainerDied","Data":"361e9b935edede4b958765527a76c4781f36ada19e305b6e2c687ceec5ed9c44"} Aug 04 00:03:05.028011 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:05.027977 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:03:05.028371 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:05.027992 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:03:05.028371 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:05.028096 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj5b" podUID="51f02f93-efb0-42f3-acaf-2de4e4db0cbc" Aug 04 00:03:05.028371 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:05.028173 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:03:05.028371 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:05.027990 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:03:05.028371 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:05.028270 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqddv" podUID="8c2ca316-6027-4942-8085-22409a8b0b81" Aug 04 00:03:05.195328 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:05.195293 2570 generic.go:356] "Generic (PLEG): container finished" podID="7866cfab-c563-4c2c-95d7-3485abec7465" containerID="63546533f8a3f641f4831aa8a03abee91f94f8be594dfd60b596789716d101c7" exitCode=0 Aug 04 00:03:05.195634 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:05.195533 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8h6f" event={"ID":"7866cfab-c563-4c2c-95d7-3485abec7465","Type":"ContainerDied","Data":"63546533f8a3f641f4831aa8a03abee91f94f8be594dfd60b596789716d101c7"} Aug 04 00:03:07.028061 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:07.028033 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:03:07.028598 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:07.028037 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:03:07.028598 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:07.028173 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:03:07.028598 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:07.028046 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:03:07.028598 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:07.028242 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj5b" podUID="51f02f93-efb0-42f3-acaf-2de4e4db0cbc" Aug 04 00:03:07.028598 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:07.028306 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqddv" podUID="8c2ca316-6027-4942-8085-22409a8b0b81" Aug 04 00:03:07.654987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:07.654910 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs\") pod \"network-metrics-daemon-9b87x\" (UID: \"e656f284-112c-4309-9151-41e5a9f45ede\") " pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:03:07.654987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:07.654946 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret\") pod \"global-pull-secret-syncer-hrj5b\" (UID: \"51f02f93-efb0-42f3-acaf-2de4e4db0cbc\") " pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:03:07.655205 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:07.655058 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:03:07.655205 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:07.655064 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:03:07.655205 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:07.655118 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret podName:51f02f93-efb0-42f3-acaf-2de4e4db0cbc nodeName:}" failed. No retries permitted until 2026-08-04 00:03:39.655098531 +0000 UTC m=+66.137764854 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret") pod "global-pull-secret-syncer-hrj5b" (UID: "51f02f93-efb0-42f3-acaf-2de4e4db0cbc") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:03:07.655205 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:07.655135 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs podName:e656f284-112c-4309-9151-41e5a9f45ede nodeName:}" failed. No retries permitted until 2026-08-04 00:03:39.655125981 +0000 UTC m=+66.137792295 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs") pod "network-metrics-daemon-9b87x" (UID: "e656f284-112c-4309-9151-41e5a9f45ede") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:03:07.856454 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:07.856424 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpq74\" (UniqueName: \"kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74\") pod \"network-check-target-cqddv\" (UID: \"8c2ca316-6027-4942-8085-22409a8b0b81\") " pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:03:07.856605 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:07.856591 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:03:07.856663 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:07.856615 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:03:07.856663 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:07.856625 2570 projected.go:196] Error preparing data for projected volume kube-api-access-mpq74 for pod openshift-network-diagnostics/network-check-target-cqddv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:03:07.856740 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:07.856685 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74 podName:8c2ca316-6027-4942-8085-22409a8b0b81 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:39.856666767 +0000 UTC m=+66.339333075 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpq74" (UniqueName: "kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74") pod "network-check-target-cqddv" (UID: "8c2ca316-6027-4942-8085-22409a8b0b81") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:03:08.804649 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.804624 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-102.ec2.internal" event="NodeReady" Aug 04 00:03:08.805021 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.804745 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 00:03:08.871736 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.871712 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fc48d6bdf-m8vf4"] Aug 04 00:03:08.876305 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.876280 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:08.878160 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.878134 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29763360-vng42"] Aug 04 00:03:08.879214 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.879195 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 04 00:03:08.879696 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.879681 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rbxcz\"" Aug 04 00:03:08.879918 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.879899 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 04 00:03:08.880424 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.880398 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 04 00:03:08.881593 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.881565 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29763360-vng42" Aug 04 00:03:08.884404 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.884340 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-rzl55\"" Aug 04 00:03:08.884404 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.884348 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\"" Aug 04 00:03:08.887591 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.887567 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 04 00:03:08.891064 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.891044 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fdgzw"] Aug 04 00:03:08.894340 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.894318 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fdgzw" Aug 04 00:03:08.894815 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.894759 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29763360-vng42"] Aug 04 00:03:08.895794 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.895777 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fc48d6bdf-m8vf4"] Aug 04 00:03:08.896574 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.896545 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-88l8k"] Aug 04 00:03:08.896665 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.896645 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 00:03:08.896735 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.896679 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 00:03:08.896735 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.896692 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n4xmt\"" Aug 04 00:03:08.896831 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.896654 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 00:03:08.899972 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.899953 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:08.902234 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.902203 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 00:03:08.902404 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.902380 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 00:03:08.902467 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.902380 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7ds4t\"" Aug 04 00:03:08.904417 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.904396 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fdgzw"] Aug 04 00:03:08.910012 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.909990 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-88l8k"] Aug 04 00:03:08.966142 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.966111 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:08.966142 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.966148 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-installation-pull-secrets\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:08.966373 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.966176 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw7nx\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-kube-api-access-cw7nx\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:08.966373 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.966260 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-certificates\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:08.966373 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.966309 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-ca-trust-extracted\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:08.966373 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.966339 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-bound-sa-token\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:08.966562 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.966388 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-image-registry-private-configuration\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:08.966562 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.966420 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-trusted-ca\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:08.966562 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.966451 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f998954-55dc-4e58-a70a-ad878047d419-serviceca\") pod \"image-pruner-29763360-vng42\" (UID: \"2f998954-55dc-4e58-a70a-ad878047d419\") " pod="openshift-image-registry/image-pruner-29763360-vng42" Aug 04 00:03:08.966562 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:08.966477 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjj54\" (UniqueName: \"kubernetes.io/projected/2f998954-55dc-4e58-a70a-ad878047d419-kube-api-access-kjj54\") pod \"image-pruner-29763360-vng42\" (UID: \"2f998954-55dc-4e58-a70a-ad878047d419\") " pod="openshift-image-registry/image-pruner-29763360-vng42" Aug 04 00:03:09.028330 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.028307 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:03:09.028330 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.028323 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:03:09.028504 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.028339 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:03:09.031338 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.031287 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 00:03:09.031338 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.031307 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qjt6l\"" Aug 04 00:03:09.031546 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.031357 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 00:03:09.031546 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.031437 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 00:03:09.031546 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.031514 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f9tct\"" Aug 04 00:03:09.031713 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.031633 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 00:03:09.067666 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.067617 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvfvd\" (UniqueName: \"kubernetes.io/projected/c3ee0892-91f7-4cab-8b96-829a660ab62d-kube-api-access-jvfvd\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:09.067764 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.067672 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:09.067764 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.067697 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-installation-pull-secrets\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:09.067764 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.067718 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw7nx\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-kube-api-access-cw7nx\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:09.067764 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.067735 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-certificates\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:09.067764 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.067758 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-ca-trust-extracted\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:09.067764 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:09.067763 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 00:03:09.068052 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:09.067779 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc48d6bdf-m8vf4: secret "image-registry-tls" not found Aug 04 00:03:09.068052 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.067783 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-bound-sa-token\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:09.068052 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.067807 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:09.068052 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:09.067833 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls podName:7a3fc7b1-2b51-4009-ade1-d8b2c609a20c nodeName:}" failed. No retries permitted until 2026-08-04 00:03:09.567812187 +0000 UTC m=+36.050478510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls") pod "image-registry-7fc48d6bdf-m8vf4" (UID: "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c") : secret "image-registry-tls" not found Aug 04 00:03:09.068052 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.067880 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9pwg\" (UniqueName: \"kubernetes.io/projected/df13c2f8-df07-47cb-82dc-2ac530b6985f-kube-api-access-d9pwg\") pod \"ingress-canary-fdgzw\" (UID: \"df13c2f8-df07-47cb-82dc-2ac530b6985f\") " pod="openshift-ingress-canary/ingress-canary-fdgzw" Aug 04 00:03:09.068052 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.067923 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-image-registry-private-configuration\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:09.068052 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.067952 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-trusted-ca\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:09.068052 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.067973 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df13c2f8-df07-47cb-82dc-2ac530b6985f-cert\") pod \"ingress-canary-fdgzw\" (UID: \"df13c2f8-df07-47cb-82dc-2ac530b6985f\") " pod="openshift-ingress-canary/ingress-canary-fdgzw" Aug 04 00:03:09.068052 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.067990 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ee0892-91f7-4cab-8b96-829a660ab62d-tmp-dir\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:09.068052 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.068021 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f998954-55dc-4e58-a70a-ad878047d419-serviceca\") pod \"image-pruner-29763360-vng42\" (UID: \"2f998954-55dc-4e58-a70a-ad878047d419\") " pod="openshift-image-registry/image-pruner-29763360-vng42" Aug 04 00:03:09.068052 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.068046 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ee0892-91f7-4cab-8b96-829a660ab62d-config-volume\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:09.068566 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.068076 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjj54\" (UniqueName: \"kubernetes.io/projected/2f998954-55dc-4e58-a70a-ad878047d419-kube-api-access-kjj54\") pod \"image-pruner-29763360-vng42\" (UID: \"2f998954-55dc-4e58-a70a-ad878047d419\") " pod="openshift-image-registry/image-pruner-29763360-vng42" Aug 04 00:03:09.068566 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.068368 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-certificates\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:09.068667 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.068647 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-ca-trust-extracted\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:09.068807 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.068781 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f998954-55dc-4e58-a70a-ad878047d419-serviceca\") pod \"image-pruner-29763360-vng42\" (UID: \"2f998954-55dc-4e58-a70a-ad878047d419\") " pod="openshift-image-registry/image-pruner-29763360-vng42" Aug 04 00:03:09.069267 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.069248 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-trusted-ca\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:09.072471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.072342 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-image-registry-private-configuration\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:09.072546 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.072348 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-installation-pull-secrets\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:09.078622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.078606 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-bound-sa-token\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:09.078692 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.078657 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw7nx\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-kube-api-access-cw7nx\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:09.078864 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.078846 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjj54\" (UniqueName: \"kubernetes.io/projected/2f998954-55dc-4e58-a70a-ad878047d419-kube-api-access-kjj54\") pod \"image-pruner-29763360-vng42\" (UID: \"2f998954-55dc-4e58-a70a-ad878047d419\") " pod="openshift-image-registry/image-pruner-29763360-vng42" Aug 04 00:03:09.168839 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.168815 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df13c2f8-df07-47cb-82dc-2ac530b6985f-cert\") pod \"ingress-canary-fdgzw\" (UID: \"df13c2f8-df07-47cb-82dc-2ac530b6985f\") " pod="openshift-ingress-canary/ingress-canary-fdgzw" Aug 04 00:03:09.168972 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.168850 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ee0892-91f7-4cab-8b96-829a660ab62d-tmp-dir\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:09.168972 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.168880 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ee0892-91f7-4cab-8b96-829a660ab62d-config-volume\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:09.169086 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.169007 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvfvd\" (UniqueName: \"kubernetes.io/projected/c3ee0892-91f7-4cab-8b96-829a660ab62d-kube-api-access-jvfvd\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:09.169135 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.169107 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:09.169182 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.169151 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9pwg\" (UniqueName: \"kubernetes.io/projected/df13c2f8-df07-47cb-82dc-2ac530b6985f-kube-api-access-d9pwg\") pod \"ingress-canary-fdgzw\" (UID: \"df13c2f8-df07-47cb-82dc-2ac530b6985f\") " pod="openshift-ingress-canary/ingress-canary-fdgzw" Aug 04 00:03:09.169283 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.169264 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ee0892-91f7-4cab-8b96-829a660ab62d-tmp-dir\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:09.169350 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:09.169272 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:09.169405 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:09.169350 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls podName:c3ee0892-91f7-4cab-8b96-829a660ab62d nodeName:}" failed. No retries permitted until 2026-08-04 00:03:09.669331739 +0000 UTC m=+36.151998052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls") pod "dns-default-88l8k" (UID: "c3ee0892-91f7-4cab-8b96-829a660ab62d") : secret "dns-default-metrics-tls" not found Aug 04 00:03:09.169478 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.169458 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ee0892-91f7-4cab-8b96-829a660ab62d-config-volume\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:09.171322 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.171304 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df13c2f8-df07-47cb-82dc-2ac530b6985f-cert\") pod \"ingress-canary-fdgzw\" (UID: \"df13c2f8-df07-47cb-82dc-2ac530b6985f\") " pod="openshift-ingress-canary/ingress-canary-fdgzw" Aug 04 00:03:09.178132 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.178113 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9pwg\" (UniqueName: \"kubernetes.io/projected/df13c2f8-df07-47cb-82dc-2ac530b6985f-kube-api-access-d9pwg\") pod \"ingress-canary-fdgzw\" (UID: \"df13c2f8-df07-47cb-82dc-2ac530b6985f\") " pod="openshift-ingress-canary/ingress-canary-fdgzw" Aug 04 00:03:09.178310 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.178291 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvfvd\" (UniqueName: \"kubernetes.io/projected/c3ee0892-91f7-4cab-8b96-829a660ab62d-kube-api-access-jvfvd\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:09.196360 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.196342 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29763360-vng42" Aug 04 00:03:09.207508 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.207492 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fdgzw" Aug 04 00:03:09.454007 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.453983 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fdgzw"] Aug 04 00:03:09.457066 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.457046 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29763360-vng42"] Aug 04 00:03:09.457778 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:03:09.457758 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf13c2f8_df07_47cb_82dc_2ac530b6985f.slice/crio-e589b3b5449fe12a8e85d6a7763988c3bd9e6d217f9b4ef700808ed4b5c5f19b WatchSource:0}: Error finding container e589b3b5449fe12a8e85d6a7763988c3bd9e6d217f9b4ef700808ed4b5c5f19b: Status 404 returned error can't find the container with id e589b3b5449fe12a8e85d6a7763988c3bd9e6d217f9b4ef700808ed4b5c5f19b Aug 04 00:03:09.460193 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:03:09.460169 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f998954_55dc_4e58_a70a_ad878047d419.slice/crio-0cff42b3d0381869b1b132db19288070646f92fb9764cf779e0c1c8c77adde2b WatchSource:0}: Error finding container 0cff42b3d0381869b1b132db19288070646f92fb9764cf779e0c1c8c77adde2b: Status 404 returned error can't find the container with id 0cff42b3d0381869b1b132db19288070646f92fb9764cf779e0c1c8c77adde2b Aug 04 00:03:09.572150 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.572110 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:09.572308 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:09.572296 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 00:03:09.572379 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:09.572314 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc48d6bdf-m8vf4: secret "image-registry-tls" not found Aug 04 00:03:09.572379 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:09.572373 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls podName:7a3fc7b1-2b51-4009-ade1-d8b2c609a20c nodeName:}" failed. No retries permitted until 2026-08-04 00:03:10.572357667 +0000 UTC m=+37.055023993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls") pod "image-registry-7fc48d6bdf-m8vf4" (UID: "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c") : secret "image-registry-tls" not found Aug 04 00:03:09.673608 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:09.673578 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:09.673775 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:09.673755 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:09.673852 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:09.673841 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls podName:c3ee0892-91f7-4cab-8b96-829a660ab62d nodeName:}" failed. No retries permitted until 2026-08-04 00:03:10.673816837 +0000 UTC m=+37.156483169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls") pod "dns-default-88l8k" (UID: "c3ee0892-91f7-4cab-8b96-829a660ab62d") : secret "dns-default-metrics-tls" not found Aug 04 00:03:10.209056 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:10.209026 2570 generic.go:356] "Generic (PLEG): container finished" podID="7866cfab-c563-4c2c-95d7-3485abec7465" containerID="2c64fc5a3430c12596412cf8b9d0f33207a0b5b4b7ca0245e0118f7c3605de50" exitCode=0 Aug 04 00:03:10.209892 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:10.209117 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8h6f" event={"ID":"7866cfab-c563-4c2c-95d7-3485abec7465","Type":"ContainerDied","Data":"2c64fc5a3430c12596412cf8b9d0f33207a0b5b4b7ca0245e0118f7c3605de50"} Aug 04 00:03:10.212773 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:10.212743 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29763360-vng42" event={"ID":"2f998954-55dc-4e58-a70a-ad878047d419","Type":"ContainerStarted","Data":"8e3f015fdf89697b6062f6ce8a3080fd52b650f4bec79ae35f58de36ded541a8"} Aug 04 00:03:10.212886 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:10.212796 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29763360-vng42" event={"ID":"2f998954-55dc-4e58-a70a-ad878047d419","Type":"ContainerStarted","Data":"0cff42b3d0381869b1b132db19288070646f92fb9764cf779e0c1c8c77adde2b"} Aug 04 00:03:10.213781 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:10.213762 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fdgzw" event={"ID":"df13c2f8-df07-47cb-82dc-2ac530b6985f","Type":"ContainerStarted","Data":"e589b3b5449fe12a8e85d6a7763988c3bd9e6d217f9b4ef700808ed4b5c5f19b"} Aug 04 00:03:10.254331 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:10.253800 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-pruner-29763360-vng42" podStartSLOduration=190.253783562 podStartE2EDuration="3m10.253783562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:10.253117308 +0000 UTC m=+36.735783639" watchObservedRunningTime="2026-08-04 00:03:10.253783562 +0000 UTC m=+36.736449893" Aug 04 00:03:10.582602 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:10.582372 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:10.582754 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:10.582516 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 00:03:10.582754 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:10.582645 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc48d6bdf-m8vf4: secret "image-registry-tls" not found Aug 04 00:03:10.582754 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:10.582705 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls podName:7a3fc7b1-2b51-4009-ade1-d8b2c609a20c nodeName:}" failed. No retries permitted until 2026-08-04 00:03:12.582686333 +0000 UTC m=+39.065352642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls") pod "image-registry-7fc48d6bdf-m8vf4" (UID: "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c") : secret "image-registry-tls" not found Aug 04 00:03:10.682950 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:10.682910 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:10.683080 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:10.683012 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:10.683080 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:10.683062 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls podName:c3ee0892-91f7-4cab-8b96-829a660ab62d nodeName:}" failed. No retries permitted until 2026-08-04 00:03:12.683045838 +0000 UTC m=+39.165712151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls") pod "dns-default-88l8k" (UID: "c3ee0892-91f7-4cab-8b96-829a660ab62d") : secret "dns-default-metrics-tls" not found Aug 04 00:03:11.219638 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:11.219554 2570 generic.go:356] "Generic (PLEG): container finished" podID="7866cfab-c563-4c2c-95d7-3485abec7465" containerID="4bb494920197fdd2414fca3f7a8ea050d03aea22121d40e408c2c48d5c95f9da" exitCode=0 Aug 04 00:03:11.220053 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:11.219641 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8h6f" event={"ID":"7866cfab-c563-4c2c-95d7-3485abec7465","Type":"ContainerDied","Data":"4bb494920197fdd2414fca3f7a8ea050d03aea22121d40e408c2c48d5c95f9da"} Aug 04 00:03:12.226188 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:12.225933 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8h6f" event={"ID":"7866cfab-c563-4c2c-95d7-3485abec7465","Type":"ContainerStarted","Data":"9788e16c73125a4f77f8287bc69ab639a0fb7855d8ffe62653bce345aeeb03d5"} Aug 04 00:03:12.227500 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:12.227475 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fdgzw" event={"ID":"df13c2f8-df07-47cb-82dc-2ac530b6985f","Type":"ContainerStarted","Data":"906392f70d29caf82148a37d28ed9fb2fb0bbc44d8aae376e4eb337fb7f2ee66"} Aug 04 00:03:12.251022 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:12.250978 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v8h6f" podStartSLOduration=9.294327316 podStartE2EDuration="38.250964674s" totalImagesPullingTime="28.956637358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:12.250392426 +0000 UTC m=+38.733058757" watchObservedRunningTime="2026-08-04 00:03:12.250964674 +0000 UTC m=+38.733631003" Aug 04 00:03:12.267972 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:12.267932 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fdgzw" podStartSLOduration=1.741394592 podStartE2EDuration="4.267919296s" totalImagesPullingTime="2.526524704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:12.267337963 +0000 UTC m=+38.750004295" watchObservedRunningTime="2026-08-04 00:03:12.267919296 +0000 UTC m=+38.750585634" Aug 04 00:03:12.502566 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:12.502535 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:12.590443 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:12.590412 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:12.595446 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:12.595427 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:12.595536 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:12.595525 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 00:03:12.595583 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:12.595538 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc48d6bdf-m8vf4: secret "image-registry-tls" not found Aug 04 00:03:12.595583 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:12.595576 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls podName:7a3fc7b1-2b51-4009-ade1-d8b2c609a20c nodeName:}" failed. No retries permitted until 2026-08-04 00:03:16.595565411 +0000 UTC m=+43.078231720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls") pod "image-registry-7fc48d6bdf-m8vf4" (UID: "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c") : secret "image-registry-tls" not found Aug 04 00:03:12.690653 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:12.690633 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:12.696692 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:12.696674 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:12.696779 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:12.696765 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:12.696824 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:12.696815 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls podName:c3ee0892-91f7-4cab-8b96-829a660ab62d nodeName:}" failed. No retries permitted until 2026-08-04 00:03:16.696801861 +0000 UTC m=+43.179468172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls") pod "dns-default-88l8k" (UID: "c3ee0892-91f7-4cab-8b96-829a660ab62d") : secret "dns-default-metrics-tls" not found Aug 04 00:03:14.053203 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:14.053163 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:14.106237 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:14.106202 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:16.627570 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:16.627528 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:16.627987 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:16.627653 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 00:03:16.627987 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:16.627666 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc48d6bdf-m8vf4: secret "image-registry-tls" not found Aug 04 00:03:16.627987 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:16.627720 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls podName:7a3fc7b1-2b51-4009-ade1-d8b2c609a20c nodeName:}" failed. No retries permitted until 2026-08-04 00:03:24.627704719 +0000 UTC m=+51.110371028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls") pod "image-registry-7fc48d6bdf-m8vf4" (UID: "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c") : secret "image-registry-tls" not found Aug 04 00:03:16.727902 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:16.727874 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:16.728037 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:16.727985 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:16.728077 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:16.728037 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls podName:c3ee0892-91f7-4cab-8b96-829a660ab62d nodeName:}" failed. No retries permitted until 2026-08-04 00:03:24.728020728 +0000 UTC m=+51.210687039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls") pod "dns-default-88l8k" (UID: "c3ee0892-91f7-4cab-8b96-829a660ab62d") : secret "dns-default-metrics-tls" not found Aug 04 00:03:17.270112 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:17.270078 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:17.333085 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:17.333048 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:22.286849 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:22.286822 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:22.388204 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:22.388175 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:23.528732 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:23.528706 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:24.681330 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:24.681296 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:24.681731 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:24.681411 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 00:03:24.681731 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:24.681425 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc48d6bdf-m8vf4: secret "image-registry-tls" not found Aug 04 00:03:24.681731 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:24.681478 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls podName:7a3fc7b1-2b51-4009-ade1-d8b2c609a20c nodeName:}" failed. No retries permitted until 2026-08-04 00:03:40.681463575 +0000 UTC m=+67.164129884 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls") pod "image-registry-7fc48d6bdf-m8vf4" (UID: "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c") : secret "image-registry-tls" not found Aug 04 00:03:24.745426 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:24.745379 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:24.781596 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:24.781567 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:24.781720 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:24.781674 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:24.781760 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:24.781732 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls podName:c3ee0892-91f7-4cab-8b96-829a660ab62d nodeName:}" failed. No retries permitted until 2026-08-04 00:03:40.781718462 +0000 UTC m=+67.264384770 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls") pod "dns-default-88l8k" (UID: "c3ee0892-91f7-4cab-8b96-829a660ab62d") : secret "dns-default-metrics-tls" not found Aug 04 00:03:24.845382 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:24.845344 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:25.655074 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:25.655043 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:25.708434 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:25.708408 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:34.205567 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:34.205540 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbj4t" Aug 04 00:03:34.354720 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:34.354690 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:34.427342 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:34.427309 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:39.674883 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:39.674849 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs\") pod \"network-metrics-daemon-9b87x\" (UID: \"e656f284-112c-4309-9151-41e5a9f45ede\") " pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:03:39.674883 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:39.674881 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret\") pod \"global-pull-secret-syncer-hrj5b\" (UID: \"51f02f93-efb0-42f3-acaf-2de4e4db0cbc\") " pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:03:39.677288 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:39.677271 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 00:03:39.677404 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:39.677389 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 00:03:39.685052 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:39.685031 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 04 00:03:39.685173 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:39.685085 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs podName:e656f284-112c-4309-9151-41e5a9f45ede nodeName:}" failed. No retries permitted until 2026-08-04 00:04:43.68507099 +0000 UTC m=+130.167737298 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs") pod "network-metrics-daemon-9b87x" (UID: "e656f284-112c-4309-9151-41e5a9f45ede") : secret "metrics-daemon-secret" not found Aug 04 00:03:39.688748 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:39.688732 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51f02f93-efb0-42f3-acaf-2de4e4db0cbc-original-pull-secret\") pod \"global-pull-secret-syncer-hrj5b\" (UID: \"51f02f93-efb0-42f3-acaf-2de4e4db0cbc\") " pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:03:39.877322 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:39.877292 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpq74\" (UniqueName: \"kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74\") pod \"network-check-target-cqddv\" (UID: \"8c2ca316-6027-4942-8085-22409a8b0b81\") " pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:03:39.879687 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:39.879662 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 00:03:39.889985 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:39.889967 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 00:03:39.900031 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:39.900013 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpq74\" (UniqueName: \"kubernetes.io/projected/8c2ca316-6027-4942-8085-22409a8b0b81-kube-api-access-mpq74\") pod \"network-check-target-cqddv\" (UID: \"8c2ca316-6027-4942-8085-22409a8b0b81\") " pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:03:39.941176 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:39.941130 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qjt6l\"" Aug 04 00:03:39.945284 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:39.945271 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj5b" Aug 04 00:03:39.949498 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:39.949484 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:03:40.070653 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:40.070627 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrj5b"] Aug 04 00:03:40.073719 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:03:40.073694 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f02f93_efb0_42f3_acaf_2de4e4db0cbc.slice/crio-8e01a6a602f24c69df7eb7de31e2d97509fccba8591e2b2bc7c7bdacbc5c5699 WatchSource:0}: Error finding container 8e01a6a602f24c69df7eb7de31e2d97509fccba8591e2b2bc7c7bdacbc5c5699: Status 404 returned error can't find the container with id 8e01a6a602f24c69df7eb7de31e2d97509fccba8591e2b2bc7c7bdacbc5c5699 Aug 04 00:03:40.083858 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:40.083839 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cqddv"] Aug 04 00:03:40.086700 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:03:40.086678 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2ca316_6027_4942_8085_22409a8b0b81.slice/crio-be6e7b3d47b88638baaf122d0037b3b63c13cf1282700467bc3c37669cf31709 WatchSource:0}: Error finding container be6e7b3d47b88638baaf122d0037b3b63c13cf1282700467bc3c37669cf31709: Status 404 returned error can't find the container with id be6e7b3d47b88638baaf122d0037b3b63c13cf1282700467bc3c37669cf31709 Aug 04 00:03:40.290151 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:40.290120 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cqddv" event={"ID":"8c2ca316-6027-4942-8085-22409a8b0b81","Type":"ContainerStarted","Data":"be6e7b3d47b88638baaf122d0037b3b63c13cf1282700467bc3c37669cf31709"} Aug 04 00:03:40.291210 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:40.291185 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrj5b" event={"ID":"51f02f93-efb0-42f3-acaf-2de4e4db0cbc","Type":"ContainerStarted","Data":"8e01a6a602f24c69df7eb7de31e2d97509fccba8591e2b2bc7c7bdacbc5c5699"} Aug 04 00:03:40.682194 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:40.682130 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:03:40.682539 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:40.682309 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 00:03:40.682539 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:40.682325 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc48d6bdf-m8vf4: secret "image-registry-tls" not found Aug 04 00:03:40.682539 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:40.682396 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls podName:7a3fc7b1-2b51-4009-ade1-d8b2c609a20c nodeName:}" failed. No retries permitted until 2026-08-04 00:04:12.68237776 +0000 UTC m=+99.165044068 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls") pod "image-registry-7fc48d6bdf-m8vf4" (UID: "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c") : secret "image-registry-tls" not found Aug 04 00:03:40.782864 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:40.782829 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:03:40.783008 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:40.782985 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:40.783065 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:03:40.783052 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls podName:c3ee0892-91f7-4cab-8b96-829a660ab62d nodeName:}" failed. No retries permitted until 2026-08-04 00:04:12.783033567 +0000 UTC m=+99.265699878 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls") pod "dns-default-88l8k" (UID: "c3ee0892-91f7-4cab-8b96-829a660ab62d") : secret "dns-default-metrics-tls" not found Aug 04 00:03:41.712596 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:41.712560 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:41.813703 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:41.813664 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:44.270541 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:44.270501 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:45.306858 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:45.306825 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrj5b" event={"ID":"51f02f93-efb0-42f3-acaf-2de4e4db0cbc","Type":"ContainerStarted","Data":"d76e646eccbbb3d623c77c12368852caade5e28b2224f442db4b69b79892b03f"} Aug 04 00:03:45.308199 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:45.308172 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cqddv" event={"ID":"8c2ca316-6027-4942-8085-22409a8b0b81","Type":"ContainerStarted","Data":"a84dab3c9880c75bf7496f2867a98778e6dff39c678645482bf2a870b10d11ad"} Aug 04 00:03:45.308333 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:45.308294 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:03:45.322136 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:45.322088 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hrj5b" podStartSLOduration=66.67516706 podStartE2EDuration="1m11.322070832s" totalImagesPullingTime="4.646903772s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:45.32173531 +0000 UTC m=+71.804401640" watchObservedRunningTime="2026-08-04 00:03:45.322070832 +0000 UTC m=+71.804737162" Aug 04 00:03:45.338517 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:45.338477 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cqddv" podStartSLOduration=66.7063197 podStartE2EDuration="1m11.338464586s" totalImagesPullingTime="4.632144886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:45.337386094 +0000 UTC m=+71.820052423" watchObservedRunningTime="2026-08-04 00:03:45.338464586 +0000 UTC m=+71.821130913" Aug 04 00:03:45.630826 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:45.630747 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:45.683084 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:45.683058 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:49.032912 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:49.032838 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:03:49.131430 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:03:49.131401 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:04:11.609852 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:04:11.609739 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:04:11.664218 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:04:11.664195 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:04:12.701608 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:04:12.701576 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:04:12.701974 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:04:12.701717 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 00:04:12.701974 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:04:12.701729 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc48d6bdf-m8vf4: secret "image-registry-tls" not found Aug 04 00:04:12.701974 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:04:12.701784 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls podName:7a3fc7b1-2b51-4009-ade1-d8b2c609a20c nodeName:}" failed. No retries permitted until 2026-08-04 00:05:16.701768918 +0000 UTC m=+163.184435225 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls") pod "image-registry-7fc48d6bdf-m8vf4" (UID: "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c") : secret "image-registry-tls" not found Aug 04 00:04:12.802608 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:04:12.802580 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:04:12.802706 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:04:12.802692 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:04:12.802770 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:04:12.802759 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls podName:c3ee0892-91f7-4cab-8b96-829a660ab62d nodeName:}" failed. No retries permitted until 2026-08-04 00:05:16.802744992 +0000 UTC m=+163.285411300 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls") pod "dns-default-88l8k" (UID: "c3ee0892-91f7-4cab-8b96-829a660ab62d") : secret "dns-default-metrics-tls" not found Aug 04 00:04:16.313848 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:04:16.313820 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cqddv" Aug 04 00:04:43.712303 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:04:43.712253 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs\") pod \"network-metrics-daemon-9b87x\" (UID: \"e656f284-112c-4309-9151-41e5a9f45ede\") " pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:04:43.712790 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:04:43.712388 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 04 00:04:43.712790 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:04:43.712452 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs podName:e656f284-112c-4309-9151-41e5a9f45ede nodeName:}" failed. No retries permitted until 2026-08-04 00:06:45.712435556 +0000 UTC m=+252.195101864 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs") pod "network-metrics-daemon-9b87x" (UID: "e656f284-112c-4309-9151-41e5a9f45ede") : secret "metrics-daemon-secret" not found Aug 04 00:05:11.890000 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:05:11.889965 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" podUID="7a3fc7b1-2b51-4009-ade1-d8b2c609a20c" Aug 04 00:05:11.913931 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:05:11.913904 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-88l8k" podUID="c3ee0892-91f7-4cab-8b96-829a660ab62d" Aug 04 00:05:12.050854 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:05:12.050820 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:05:12.520721 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:05:12.520696 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:05:12.520877 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:05:12.520696 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-88l8k" Aug 04 00:05:16.736105 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:05:16.736062 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:05:16.736576 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:05:16.736189 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 00:05:16.736576 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:05:16.736203 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc48d6bdf-m8vf4: secret "image-registry-tls" not found Aug 04 00:05:16.736576 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:05:16.736302 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls podName:7a3fc7b1-2b51-4009-ade1-d8b2c609a20c nodeName:}" failed. No retries permitted until 2026-08-04 00:07:18.736283854 +0000 UTC m=+285.218950161 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls") pod "image-registry-7fc48d6bdf-m8vf4" (UID: "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c") : secret "image-registry-tls" not found Aug 04 00:05:16.836855 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:05:16.836821 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:05:16.837043 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:05:16.836976 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:05:16.837108 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:05:16.837064 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls podName:c3ee0892-91f7-4cab-8b96-829a660ab62d nodeName:}" failed. No retries permitted until 2026-08-04 00:07:18.837048766 +0000 UTC m=+285.319715074 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls") pod "dns-default-88l8k" (UID: "c3ee0892-91f7-4cab-8b96-829a660ab62d") : secret "dns-default-metrics-tls" not found Aug 04 00:05:23.028190 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:05:23.028145 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:05:28.866364 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:05:28.866277 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:34.274282 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.274242 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z"] Aug 04 00:06:34.276001 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.275979 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z" Aug 04 00:06:34.277765 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.277739 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-2km27"] Aug 04 00:06:34.278750 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.278726 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Aug 04 00:06:34.278919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.278848 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Aug 04 00:06:34.278919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.278860 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-smk9c\"" Aug 04 00:06:34.279523 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.279498 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2km27" Aug 04 00:06:34.279689 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.279671 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Aug 04 00:06:34.279776 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.279680 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Aug 04 00:06:34.282141 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.282096 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5228b\"" Aug 04 00:06:34.292689 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.292666 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z"] Aug 04 00:06:34.297422 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.297398 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-2km27"] Aug 04 00:06:34.369082 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.369055 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-clngv"] Aug 04 00:06:34.370643 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.370624 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-clngv" Aug 04 00:06:34.373094 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.373071 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Aug 04 00:06:34.373897 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.373751 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl"] Aug 04 00:06:34.375298 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.375273 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6d5hp\"" Aug 04 00:06:34.375503 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.375486 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Aug 04 00:06:34.376097 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.376073 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp"] Aug 04 00:06:34.376197 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.376186 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" Aug 04 00:06:34.377810 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.377790 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-mhqk6"] Aug 04 00:06:34.377921 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.377904 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp" Aug 04 00:06:34.378451 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.378428 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 00:06:34.379505 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.379489 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-7b4h2"] Aug 04 00:06:34.379622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.379603 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:34.379890 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.379790 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 00:06:34.380770 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.380382 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nrllc\"" Aug 04 00:06:34.380882 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.380172 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Aug 04 00:06:34.385236 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.382173 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Aug 04 00:06:34.385236 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.382537 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-s882n\"" Aug 04 00:06:34.385236 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.382807 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Aug 04 00:06:34.385236 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.383437 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Aug 04 00:06:34.385236 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.384124 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Aug 04 00:06:34.385236 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.384141 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Aug 04 00:06:34.385236 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.384755 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Aug 04 00:06:34.387585 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.387523 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-94wgh\"" Aug 04 00:06:34.387893 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.387873 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Aug 04 00:06:34.389020 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.388999 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Aug 04 00:06:34.390531 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.390493 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-clngv"] Aug 04 00:06:34.390633 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.390565 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.390633 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.390621 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl"] Aug 04 00:06:34.391599 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.391582 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp"] Aug 04 00:06:34.392620 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.392596 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 00:06:34.392806 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.392755 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-mhqk6"] Aug 04 00:06:34.392967 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.392949 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Aug 04 00:06:34.393113 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.393091 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bv9v9\"" Aug 04 00:06:34.393266 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.393217 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 00:06:34.393358 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.393343 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Aug 04 00:06:34.393812 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.393793 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-7b4h2"] Aug 04 00:06:34.395964 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.395945 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Aug 04 00:06:34.398554 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.398531 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Aug 04 00:06:34.447616 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.447598 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mzkt\" (UniqueName: \"kubernetes.io/projected/68bf8531-7950-4913-8b4b-ada9c386a6a3-kube-api-access-6mzkt\") pod \"service-ca-operator-784f97bcf6-z2j5z\" (UID: \"68bf8531-7950-4913-8b4b-ada9c386a6a3\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z" Aug 04 00:06:34.447725 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.447629 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7l6p\" (UniqueName: \"kubernetes.io/projected/f1fc2293-e082-4b4c-9ffd-e1ddfd67ca55-kube-api-access-l7l6p\") pod \"network-check-source-65bdbbff45-2km27\" (UID: \"f1fc2293-e082-4b4c-9ffd-e1ddfd67ca55\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2km27" Aug 04 00:06:34.447725 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.447648 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68bf8531-7950-4913-8b4b-ada9c386a6a3-config\") pod \"service-ca-operator-784f97bcf6-z2j5z\" (UID: \"68bf8531-7950-4913-8b4b-ada9c386a6a3\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z" Aug 04 00:06:34.447725 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.447672 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68bf8531-7950-4913-8b4b-ada9c386a6a3-serving-cert\") pod \"service-ca-operator-784f97bcf6-z2j5z\" (UID: \"68bf8531-7950-4913-8b4b-ada9c386a6a3\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z" Aug 04 00:06:34.447725 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.447703 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7lsr\" (UniqueName: \"kubernetes.io/projected/628bb663-617f-4999-977b-a02ca26c1b19-kube-api-access-d7lsr\") pod \"volume-data-source-validator-5b8776669-clngv\" (UID: \"628bb663-617f-4999-977b-a02ca26c1b19\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-clngv" Aug 04 00:06:34.548558 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.548500 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7l6p\" (UniqueName: \"kubernetes.io/projected/f1fc2293-e082-4b4c-9ffd-e1ddfd67ca55-kube-api-access-l7l6p\") pod \"network-check-source-65bdbbff45-2km27\" (UID: \"f1fc2293-e082-4b4c-9ffd-e1ddfd67ca55\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2km27" Aug 04 00:06:34.548558 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.548535 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7lsr\" (UniqueName: \"kubernetes.io/projected/628bb663-617f-4999-977b-a02ca26c1b19-kube-api-access-d7lsr\") pod \"volume-data-source-validator-5b8776669-clngv\" (UID: \"628bb663-617f-4999-977b-a02ca26c1b19\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-clngv" Aug 04 00:06:34.548676 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.548566 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/036e01c1-b816-4ff5-8605-1ab0a192f4ee-trusted-ca\") pod \"console-operator-5ccd6b445f-mhqk6\" (UID: \"036e01c1-b816-4ff5-8605-1ab0a192f4ee\") " pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:34.548676 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.548629 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/036e01c1-b816-4ff5-8605-1ab0a192f4ee-serving-cert\") pod \"console-operator-5ccd6b445f-mhqk6\" (UID: \"036e01c1-b816-4ff5-8605-1ab0a192f4ee\") " pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:34.548676 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.548652 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbfsl\" (UID: \"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" Aug 04 00:06:34.548767 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.548678 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68bf8531-7950-4913-8b4b-ada9c386a6a3-serving-cert\") pod \"service-ca-operator-784f97bcf6-z2j5z\" (UID: \"68bf8531-7950-4913-8b4b-ada9c386a6a3\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z" Aug 04 00:06:34.548767 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.548743 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-tmp\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.548827 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.548777 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-service-ca-bundle\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.548827 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.548794 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-serving-cert\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.548827 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.548812 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-snapshots\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.548933 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.548836 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-49dhp\" (UID: \"e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp" Aug 04 00:06:34.548933 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.548852 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-vbfsl\" (UID: \"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" Aug 04 00:06:34.548933 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.548913 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68bf8531-7950-4913-8b4b-ada9c386a6a3-config\") pod \"service-ca-operator-784f97bcf6-z2j5z\" (UID: \"68bf8531-7950-4913-8b4b-ada9c386a6a3\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z" Aug 04 00:06:34.549070 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.548937 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5lvh\" (UniqueName: \"kubernetes.io/projected/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-kube-api-access-z5lvh\") pod \"cluster-samples-operator-559cbf5745-49dhp\" (UID: \"e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp" Aug 04 00:06:34.549070 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.549035 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6l95\" (UniqueName: \"kubernetes.io/projected/036e01c1-b816-4ff5-8605-1ab0a192f4ee-kube-api-access-x6l95\") pod \"console-operator-5ccd6b445f-mhqk6\" (UID: \"036e01c1-b816-4ff5-8605-1ab0a192f4ee\") " pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:34.549070 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.549058 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/036e01c1-b816-4ff5-8605-1ab0a192f4ee-config\") pod \"console-operator-5ccd6b445f-mhqk6\" (UID: \"036e01c1-b816-4ff5-8605-1ab0a192f4ee\") " pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:34.549244 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.549083 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hr6r\" (UniqueName: \"kubernetes.io/projected/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-kube-api-access-6hr6r\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.549244 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.549101 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcv47\" (UniqueName: \"kubernetes.io/projected/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-kube-api-access-vcv47\") pod \"cluster-monitoring-operator-6f55c4f58d-vbfsl\" (UID: \"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" Aug 04 00:06:34.549244 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.549132 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mzkt\" (UniqueName: \"kubernetes.io/projected/68bf8531-7950-4913-8b4b-ada9c386a6a3-kube-api-access-6mzkt\") pod \"service-ca-operator-784f97bcf6-z2j5z\" (UID: \"68bf8531-7950-4913-8b4b-ada9c386a6a3\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z" Aug 04 00:06:34.549244 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.549176 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-trusted-ca-bundle\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.549504 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.549484 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68bf8531-7950-4913-8b4b-ada9c386a6a3-config\") pod \"service-ca-operator-784f97bcf6-z2j5z\" (UID: \"68bf8531-7950-4913-8b4b-ada9c386a6a3\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z" Aug 04 00:06:34.550848 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.550830 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68bf8531-7950-4913-8b4b-ada9c386a6a3-serving-cert\") pod \"service-ca-operator-784f97bcf6-z2j5z\" (UID: \"68bf8531-7950-4913-8b4b-ada9c386a6a3\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z" Aug 04 00:06:34.556404 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.556376 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mzkt\" (UniqueName: \"kubernetes.io/projected/68bf8531-7950-4913-8b4b-ada9c386a6a3-kube-api-access-6mzkt\") pod \"service-ca-operator-784f97bcf6-z2j5z\" (UID: \"68bf8531-7950-4913-8b4b-ada9c386a6a3\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z" Aug 04 00:06:34.556483 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.556384 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7lsr\" (UniqueName: \"kubernetes.io/projected/628bb663-617f-4999-977b-a02ca26c1b19-kube-api-access-d7lsr\") pod \"volume-data-source-validator-5b8776669-clngv\" (UID: \"628bb663-617f-4999-977b-a02ca26c1b19\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-clngv" Aug 04 00:06:34.556526 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.556500 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7l6p\" (UniqueName: \"kubernetes.io/projected/f1fc2293-e082-4b4c-9ffd-e1ddfd67ca55-kube-api-access-l7l6p\") pod \"network-check-source-65bdbbff45-2km27\" (UID: \"f1fc2293-e082-4b4c-9ffd-e1ddfd67ca55\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2km27" Aug 04 00:06:34.587438 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.587416 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z" Aug 04 00:06:34.592550 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.592534 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2km27" Aug 04 00:06:34.593510 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.593487 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:34.644945 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.644864 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:34.650115 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.650086 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/036e01c1-b816-4ff5-8605-1ab0a192f4ee-trusted-ca\") pod \"console-operator-5ccd6b445f-mhqk6\" (UID: \"036e01c1-b816-4ff5-8605-1ab0a192f4ee\") " pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:34.650260 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.650171 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/036e01c1-b816-4ff5-8605-1ab0a192f4ee-serving-cert\") pod \"console-operator-5ccd6b445f-mhqk6\" (UID: \"036e01c1-b816-4ff5-8605-1ab0a192f4ee\") " pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:34.650260 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.650201 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbfsl\" (UID: \"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" Aug 04 00:06:34.650375 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.650278 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-tmp\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.650375 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.650308 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-service-ca-bundle\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.650375 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.650332 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-serving-cert\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.650375 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.650368 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-snapshots\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.650572 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.650400 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-49dhp\" (UID: \"e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp" Aug 04 00:06:34.650572 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.650427 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-vbfsl\" (UID: \"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" Aug 04 00:06:34.650572 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.650475 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5lvh\" (UniqueName: \"kubernetes.io/projected/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-kube-api-access-z5lvh\") pod \"cluster-samples-operator-559cbf5745-49dhp\" (UID: \"e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp" Aug 04 00:06:34.650572 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.650509 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6l95\" (UniqueName: \"kubernetes.io/projected/036e01c1-b816-4ff5-8605-1ab0a192f4ee-kube-api-access-x6l95\") pod \"console-operator-5ccd6b445f-mhqk6\" (UID: \"036e01c1-b816-4ff5-8605-1ab0a192f4ee\") " pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:34.650572 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.650537 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/036e01c1-b816-4ff5-8605-1ab0a192f4ee-config\") pod \"console-operator-5ccd6b445f-mhqk6\" (UID: \"036e01c1-b816-4ff5-8605-1ab0a192f4ee\") " pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:34.650803 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.650575 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hr6r\" (UniqueName: \"kubernetes.io/projected/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-kube-api-access-6hr6r\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.650803 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.650610 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcv47\" (UniqueName: \"kubernetes.io/projected/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-kube-api-access-vcv47\") pod \"cluster-monitoring-operator-6f55c4f58d-vbfsl\" (UID: \"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" Aug 04 00:06:34.650803 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.650636 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-trusted-ca-bundle\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.653960 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:34.651386 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 00:06:34.653960 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:34.651469 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls podName:3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:35.151448864 +0000 UTC m=+241.634115265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vbfsl" (UID: "3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5") : secret "cluster-monitoring-operator-tls" not found Aug 04 00:06:34.653960 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.651777 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-trusted-ca-bundle\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.653960 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:34.651904 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 00:06:34.653960 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:34.651959 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls podName:e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb nodeName:}" failed. No retries permitted until 2026-08-04 00:06:35.151941979 +0000 UTC m=+241.634608290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-49dhp" (UID: "e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb") : secret "samples-operator-tls" not found Aug 04 00:06:34.653960 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.652684 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-vbfsl\" (UID: \"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" Aug 04 00:06:34.653960 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.653110 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/036e01c1-b816-4ff5-8605-1ab0a192f4ee-trusted-ca\") pod \"console-operator-5ccd6b445f-mhqk6\" (UID: \"036e01c1-b816-4ff5-8605-1ab0a192f4ee\") " pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:34.653960 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.653790 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-snapshots\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.653960 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.653799 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/036e01c1-b816-4ff5-8605-1ab0a192f4ee-config\") pod \"console-operator-5ccd6b445f-mhqk6\" (UID: \"036e01c1-b816-4ff5-8605-1ab0a192f4ee\") " pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:34.653960 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.653831 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-tmp\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.654541 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.654253 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-service-ca-bundle\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.656005 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.655709 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/036e01c1-b816-4ff5-8605-1ab0a192f4ee-serving-cert\") pod \"console-operator-5ccd6b445f-mhqk6\" (UID: \"036e01c1-b816-4ff5-8605-1ab0a192f4ee\") " pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:34.657247 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.657128 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-serving-cert\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.660986 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.660934 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5lvh\" (UniqueName: \"kubernetes.io/projected/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-kube-api-access-z5lvh\") pod \"cluster-samples-operator-559cbf5745-49dhp\" (UID: \"e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp" Aug 04 00:06:34.660986 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.660934 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6l95\" (UniqueName: \"kubernetes.io/projected/036e01c1-b816-4ff5-8605-1ab0a192f4ee-kube-api-access-x6l95\") pod \"console-operator-5ccd6b445f-mhqk6\" (UID: \"036e01c1-b816-4ff5-8605-1ab0a192f4ee\") " pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:34.661155 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.661085 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hr6r\" (UniqueName: \"kubernetes.io/projected/f076d5df-065e-4fa3-b3c2-341bd58b1c1d-kube-api-access-6hr6r\") pod \"insights-operator-77696685b-7b4h2\" (UID: \"f076d5df-065e-4fa3-b3c2-341bd58b1c1d\") " pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.661476 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.661452 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcv47\" (UniqueName: \"kubernetes.io/projected/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-kube-api-access-vcv47\") pod \"cluster-monitoring-operator-6f55c4f58d-vbfsl\" (UID: \"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" Aug 04 00:06:34.679601 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.679580 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-clngv" Aug 04 00:06:34.705581 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.705558 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:34.710407 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.710383 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-7b4h2" Aug 04 00:06:34.714507 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.714482 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z"] Aug 04 00:06:34.720193 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:06:34.720166 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68bf8531_7950_4913_8b4b_ada9c386a6a3.slice/crio-2f09cd4a46b8db948f062752e809fc10be03be46efb9373b7ba63760056ea3f4 WatchSource:0}: Error finding container 2f09cd4a46b8db948f062752e809fc10be03be46efb9373b7ba63760056ea3f4: Status 404 returned error can't find the container with id 2f09cd4a46b8db948f062752e809fc10be03be46efb9373b7ba63760056ea3f4 Aug 04 00:06:34.730685 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.730662 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-2km27"] Aug 04 00:06:34.733486 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:06:34.733460 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fc2293_e082_4b4c_9ffd_e1ddfd67ca55.slice/crio-65826d177a53ceac975a1d744db466e86959d70b8b056e788b6828ec76a092f4 WatchSource:0}: Error finding container 65826d177a53ceac975a1d744db466e86959d70b8b056e788b6828ec76a092f4: Status 404 returned error can't find the container with id 65826d177a53ceac975a1d744db466e86959d70b8b056e788b6828ec76a092f4 Aug 04 00:06:34.831304 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.831238 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-clngv"] Aug 04 00:06:34.834754 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:06:34.834730 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628bb663_617f_4999_977b_a02ca26c1b19.slice/crio-a31eb3a13b8fe264d0d295eceb94c9a5d6da0f63f869d3da16fe9b54116cbd22 WatchSource:0}: Error finding container a31eb3a13b8fe264d0d295eceb94c9a5d6da0f63f869d3da16fe9b54116cbd22: Status 404 returned error can't find the container with id a31eb3a13b8fe264d0d295eceb94c9a5d6da0f63f869d3da16fe9b54116cbd22 Aug 04 00:06:34.852196 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.852171 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-7b4h2"] Aug 04 00:06:34.873704 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:34.873681 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-mhqk6"] Aug 04 00:06:34.876182 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:06:34.876154 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036e01c1_b816_4ff5_8605_1ab0a192f4ee.slice/crio-73ae22c4d71aeb84419d97fa03f7f706e8081f8645e1acc04b6424f42348825c WatchSource:0}: Error finding container 73ae22c4d71aeb84419d97fa03f7f706e8081f8645e1acc04b6424f42348825c: Status 404 returned error can't find the container with id 73ae22c4d71aeb84419d97fa03f7f706e8081f8645e1acc04b6424f42348825c Aug 04 00:06:35.156289 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:35.156217 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbfsl\" (UID: \"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" Aug 04 00:06:35.156289 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:35.156282 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-49dhp\" (UID: \"e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp" Aug 04 00:06:35.156412 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:35.156360 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 00:06:35.156412 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:35.156375 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 00:06:35.156491 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:35.156413 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls podName:3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:36.156396654 +0000 UTC m=+242.639062963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vbfsl" (UID: "3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5") : secret "cluster-monitoring-operator-tls" not found Aug 04 00:06:35.156491 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:35.156427 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls podName:e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb nodeName:}" failed. No retries permitted until 2026-08-04 00:06:36.156421014 +0000 UTC m=+242.639087322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-49dhp" (UID: "e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb") : secret "samples-operator-tls" not found Aug 04 00:06:35.728135 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:35.728098 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" event={"ID":"036e01c1-b816-4ff5-8605-1ab0a192f4ee","Type":"ContainerStarted","Data":"73ae22c4d71aeb84419d97fa03f7f706e8081f8645e1acc04b6424f42348825c"} Aug 04 00:06:35.729956 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:35.729880 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z" event={"ID":"68bf8531-7950-4913-8b4b-ada9c386a6a3","Type":"ContainerStarted","Data":"2f09cd4a46b8db948f062752e809fc10be03be46efb9373b7ba63760056ea3f4"} Aug 04 00:06:35.731956 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:35.731907 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-7b4h2" event={"ID":"f076d5df-065e-4fa3-b3c2-341bd58b1c1d","Type":"ContainerStarted","Data":"3f2036853370210555fdda51957a7e44e6afd40c86a3f396cb0637a523162af5"} Aug 04 00:06:35.733971 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:35.733906 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-clngv" event={"ID":"628bb663-617f-4999-977b-a02ca26c1b19","Type":"ContainerStarted","Data":"a31eb3a13b8fe264d0d295eceb94c9a5d6da0f63f869d3da16fe9b54116cbd22"} Aug 04 00:06:35.736869 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:35.736174 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2km27" event={"ID":"f1fc2293-e082-4b4c-9ffd-e1ddfd67ca55","Type":"ContainerStarted","Data":"228220e15010379f14c18bfe46bb65abff2de39a7feda58a57ef48865a88fa20"} Aug 04 00:06:35.736869 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:35.736210 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2km27" event={"ID":"f1fc2293-e082-4b4c-9ffd-e1ddfd67ca55","Type":"ContainerStarted","Data":"65826d177a53ceac975a1d744db466e86959d70b8b056e788b6828ec76a092f4"} Aug 04 00:06:36.166288 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:36.166003 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbfsl\" (UID: \"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" Aug 04 00:06:36.166288 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:36.166083 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-49dhp\" (UID: \"e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp" Aug 04 00:06:36.166288 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:36.166133 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 00:06:36.166288 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:36.166196 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls podName:3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:38.166179005 +0000 UTC m=+244.648845314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vbfsl" (UID: "3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5") : secret "cluster-monitoring-operator-tls" not found Aug 04 00:06:36.166288 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:36.166197 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 00:06:36.166288 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:36.166266 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls podName:e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb nodeName:}" failed. No retries permitted until 2026-08-04 00:06:38.166249948 +0000 UTC m=+244.648916258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-49dhp" (UID: "e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb") : secret "samples-operator-tls" not found Aug 04 00:06:36.741171 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:36.741135 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-clngv" event={"ID":"628bb663-617f-4999-977b-a02ca26c1b19","Type":"ContainerStarted","Data":"eee60b1d08eb9f910b38ae5a66b7e9c95b50d1ba4120f2a3d6a109563f566412"} Aug 04 00:06:36.756019 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:36.755977 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-clngv" podStartSLOduration=1.587610789 podStartE2EDuration="2.755961534s" totalImagesPullingTime="1.168350745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:06:36.755616475 +0000 UTC m=+243.238282826" watchObservedRunningTime="2026-08-04 00:06:36.755961534 +0000 UTC m=+243.238627845" Aug 04 00:06:36.756443 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:36.756415 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-2km27" podStartSLOduration=2.7564061349999998 podStartE2EDuration="2.756406135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:06:35.752718019 +0000 UTC m=+242.235384350" watchObservedRunningTime="2026-08-04 00:06:36.756406135 +0000 UTC m=+243.239072466" Aug 04 00:06:36.837189 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:36.837160 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:36.940380 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:36.940311 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:37.834559 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:37.834520 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:37.926629 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:37.926596 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:38.026318 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:38.026280 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:38.183815 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:38.183782 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbfsl\" (UID: \"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" Aug 04 00:06:38.183929 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:38.183835 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-49dhp\" (UID: \"e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp" Aug 04 00:06:38.183995 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:38.183927 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 00:06:38.183995 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:38.183944 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 00:06:38.183995 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:38.183988 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls podName:3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:42.183973601 +0000 UTC m=+248.666639909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vbfsl" (UID: "3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5") : secret "cluster-monitoring-operator-tls" not found Aug 04 00:06:38.184140 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:38.184002 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls podName:e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb nodeName:}" failed. No retries permitted until 2026-08-04 00:06:42.183996259 +0000 UTC m=+248.666662566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-49dhp" (UID: "e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb") : secret "samples-operator-tls" not found Aug 04 00:06:38.749670 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:38.749643 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/0.log" Aug 04 00:06:38.749810 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:38.749686 2570 generic.go:356] "Generic (PLEG): container finished" podID="036e01c1-b816-4ff5-8605-1ab0a192f4ee" containerID="3354993e1287f1749cd5eed946034374336b437572376c46bcc1468a079ee0ed" exitCode=255 Aug 04 00:06:38.749810 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:38.749766 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" event={"ID":"036e01c1-b816-4ff5-8605-1ab0a192f4ee","Type":"ContainerDied","Data":"3354993e1287f1749cd5eed946034374336b437572376c46bcc1468a079ee0ed"} Aug 04 00:06:38.749964 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:38.749948 2570 scope.go:117] "RemoveContainer" containerID="3354993e1287f1749cd5eed946034374336b437572376c46bcc1468a079ee0ed" Aug 04 00:06:38.751278 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:38.751253 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z" event={"ID":"68bf8531-7950-4913-8b4b-ada9c386a6a3","Type":"ContainerStarted","Data":"ed4b6e28f010064a8a72013f9cde36d57ad70dfa4cd9fdfcf427dd0436f4134d"} Aug 04 00:06:38.757943 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:38.757915 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-7b4h2" event={"ID":"f076d5df-065e-4fa3-b3c2-341bd58b1c1d","Type":"ContainerStarted","Data":"f2ffa648e668974053596a148e601f22d327b18f9372b3b9f26e741082580003"} Aug 04 00:06:38.786001 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:38.785948 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-7b4h2" podStartSLOduration=1.6807944510000001 podStartE2EDuration="4.785933851s" totalImagesPullingTime="3.1051394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:06:38.784571878 +0000 UTC m=+245.267238200" watchObservedRunningTime="2026-08-04 00:06:38.785933851 +0000 UTC m=+245.268600175" Aug 04 00:06:38.800167 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:38.799926 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z" podStartSLOduration=1.559769057 podStartE2EDuration="4.799910692s" totalImagesPullingTime="3.240141635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:06:38.799496139 +0000 UTC m=+245.282162474" watchObservedRunningTime="2026-08-04 00:06:38.799910692 +0000 UTC m=+245.282577023" Aug 04 00:06:39.142268 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.142184 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cx6wf"] Aug 04 00:06:39.144137 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.144111 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:39.146841 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.146815 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 00:06:39.146841 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.146831 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 00:06:39.147023 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.146888 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pbnpj\"" Aug 04 00:06:39.155879 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.155854 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cx6wf"] Aug 04 00:06:39.182542 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.182514 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:39.241818 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.241798 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:39.293862 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.293840 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:39.293946 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.293871 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74a07772-f95e-453a-8694-1ccce1cf23ac-crio-socket\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:39.293946 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.293904 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/74a07772-f95e-453a-8694-1ccce1cf23ac-data-volume\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:39.294040 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.293990 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:39.294040 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.294015 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq5r6\" (UniqueName: \"kubernetes.io/projected/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-api-access-nq5r6\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:39.394493 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.394433 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:39.394493 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.394473 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq5r6\" (UniqueName: \"kubernetes.io/projected/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-api-access-nq5r6\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:39.394652 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.394495 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:39.394652 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:39.394498 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 00:06:39.394652 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.394514 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74a07772-f95e-453a-8694-1ccce1cf23ac-crio-socket\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:39.394652 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:39.394592 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 00:06:39.394652 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.394599 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74a07772-f95e-453a-8694-1ccce1cf23ac-crio-socket\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:39.394652 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:39.394637 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls podName:74a07772-f95e-453a-8694-1ccce1cf23ac nodeName:}" failed. No retries permitted until 2026-08-04 00:06:39.894620516 +0000 UTC m=+246.377286854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cx6wf" (UID: "74a07772-f95e-453a-8694-1ccce1cf23ac") : secret "insights-runtime-extractor-tls" not found Aug 04 00:06:39.394947 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.394657 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/74a07772-f95e-453a-8694-1ccce1cf23ac-data-volume\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:39.394947 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:39.394734 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm podName:74a07772-f95e-453a-8694-1ccce1cf23ac nodeName:}" failed. No retries permitted until 2026-08-04 00:06:39.894721397 +0000 UTC m=+246.377387709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cx6wf" (UID: "74a07772-f95e-453a-8694-1ccce1cf23ac") : configmap "kube-rbac-proxy" not found Aug 04 00:06:39.394947 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.394918 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/74a07772-f95e-453a-8694-1ccce1cf23ac-data-volume\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:39.404786 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.404766 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq5r6\" (UniqueName: \"kubernetes.io/projected/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-api-access-nq5r6\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:39.716243 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.716168 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:39.762071 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.762049 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:06:39.762464 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.762450 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/0.log" Aug 04 00:06:39.762520 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.762483 2570 generic.go:356] "Generic (PLEG): container finished" podID="036e01c1-b816-4ff5-8605-1ab0a192f4ee" containerID="ea5ff1a07b6bcc413a0f3d641fb02814d543f534217edf567d8f9e9d57f5add8" exitCode=255 Aug 04 00:06:39.762567 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.762548 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" event={"ID":"036e01c1-b816-4ff5-8605-1ab0a192f4ee","Type":"ContainerDied","Data":"ea5ff1a07b6bcc413a0f3d641fb02814d543f534217edf567d8f9e9d57f5add8"} Aug 04 00:06:39.762615 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.762603 2570 scope.go:117] "RemoveContainer" containerID="3354993e1287f1749cd5eed946034374336b437572376c46bcc1468a079ee0ed" Aug 04 00:06:39.762837 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.762810 2570 scope.go:117] "RemoveContainer" containerID="ea5ff1a07b6bcc413a0f3d641fb02814d543f534217edf567d8f9e9d57f5add8" Aug 04 00:06:39.763032 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:39.763005 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-mhqk6_openshift-console-operator(036e01c1-b816-4ff5-8605-1ab0a192f4ee)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" podUID="036e01c1-b816-4ff5-8605-1ab0a192f4ee" Aug 04 00:06:39.782005 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.781984 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:39.898972 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.898927 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:39.899085 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:39.899025 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:39.899147 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:39.899096 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 00:06:39.899147 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:39.899125 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 00:06:39.899267 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:39.899166 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm podName:74a07772-f95e-453a-8694-1ccce1cf23ac nodeName:}" failed. No retries permitted until 2026-08-04 00:06:40.899147885 +0000 UTC m=+247.381814193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cx6wf" (UID: "74a07772-f95e-453a-8694-1ccce1cf23ac") : configmap "kube-rbac-proxy" not found Aug 04 00:06:39.899359 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:39.899317 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls podName:74a07772-f95e-453a-8694-1ccce1cf23ac nodeName:}" failed. No retries permitted until 2026-08-04 00:06:40.899281901 +0000 UTC m=+247.381948223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cx6wf" (UID: "74a07772-f95e-453a-8694-1ccce1cf23ac") : secret "insights-runtime-extractor-tls" not found Aug 04 00:06:40.013388 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:40.013366 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:40.062016 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:40.061993 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:40.223926 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:40.223901 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:40.323732 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:40.323679 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:40.693837 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:40.693773 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:40.767997 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:40.767977 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:06:40.768340 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:40.768323 2570 scope.go:117] "RemoveContainer" containerID="ea5ff1a07b6bcc413a0f3d641fb02814d543f534217edf567d8f9e9d57f5add8" Aug 04 00:06:40.768495 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:40.768475 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-mhqk6_openshift-console-operator(036e01c1-b816-4ff5-8605-1ab0a192f4ee)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" podUID="036e01c1-b816-4ff5-8605-1ab0a192f4ee" Aug 04 00:06:40.907359 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:40.907335 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:40.907461 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:40.907391 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:40.907525 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:40.907475 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 00:06:40.907578 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:40.907525 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 00:06:40.907578 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:40.907547 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm podName:74a07772-f95e-453a-8694-1ccce1cf23ac nodeName:}" failed. No retries permitted until 2026-08-04 00:06:42.90752995 +0000 UTC m=+249.390196258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cx6wf" (UID: "74a07772-f95e-453a-8694-1ccce1cf23ac") : configmap "kube-rbac-proxy" not found Aug 04 00:06:40.907692 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:40.907586 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls podName:74a07772-f95e-453a-8694-1ccce1cf23ac nodeName:}" failed. No retries permitted until 2026-08-04 00:06:42.907569328 +0000 UTC m=+249.390235643 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cx6wf" (UID: "74a07772-f95e-453a-8694-1ccce1cf23ac") : secret "insights-runtime-extractor-tls" not found Aug 04 00:06:41.886436 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:41.886398 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-bns6q"] Aug 04 00:06:41.890847 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:41.890823 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-bns6q" Aug 04 00:06:41.893526 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:41.893507 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Aug 04 00:06:41.893633 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:41.893528 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Aug 04 00:06:41.894718 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:41.894702 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Aug 04 00:06:41.894818 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:41.894738 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Aug 04 00:06:41.894881 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:41.894822 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6bmwh\"" Aug 04 00:06:41.898892 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:41.898871 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-bns6q"] Aug 04 00:06:41.918849 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:41.918831 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mbkx7_fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117/dns-node-resolver/0.log" Aug 04 00:06:41.932101 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:41.932082 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:42.015207 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.015177 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lwzg\" (UniqueName: \"kubernetes.io/projected/c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1-kube-api-access-9lwzg\") pod \"service-ca-98c88fbb5-bns6q\" (UID: \"c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-bns6q" Aug 04 00:06:42.015392 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.015241 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1-signing-key\") pod \"service-ca-98c88fbb5-bns6q\" (UID: \"c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-bns6q" Aug 04 00:06:42.015392 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.015304 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1-signing-cabundle\") pod \"service-ca-98c88fbb5-bns6q\" (UID: \"c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-bns6q" Aug 04 00:06:42.115823 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.115798 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lwzg\" (UniqueName: \"kubernetes.io/projected/c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1-kube-api-access-9lwzg\") pod \"service-ca-98c88fbb5-bns6q\" (UID: \"c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-bns6q" Aug 04 00:06:42.115930 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.115840 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1-signing-key\") pod \"service-ca-98c88fbb5-bns6q\" (UID: \"c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-bns6q" Aug 04 00:06:42.115930 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.115866 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1-signing-cabundle\") pod \"service-ca-98c88fbb5-bns6q\" (UID: \"c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-bns6q" Aug 04 00:06:42.116450 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.116431 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1-signing-cabundle\") pod \"service-ca-98c88fbb5-bns6q\" (UID: \"c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-bns6q" Aug 04 00:06:42.118308 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.118286 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1-signing-key\") pod \"service-ca-98c88fbb5-bns6q\" (UID: \"c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-bns6q" Aug 04 00:06:42.123574 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.123554 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lwzg\" (UniqueName: \"kubernetes.io/projected/c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1-kube-api-access-9lwzg\") pod \"service-ca-98c88fbb5-bns6q\" (UID: \"c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-bns6q" Aug 04 00:06:42.200579 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.200532 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-bns6q" Aug 04 00:06:42.216361 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.216337 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbfsl\" (UID: \"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" Aug 04 00:06:42.216459 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.216399 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-49dhp\" (UID: \"e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp" Aug 04 00:06:42.216524 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:42.216466 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 00:06:42.216524 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:42.216521 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls podName:3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:50.216507563 +0000 UTC m=+256.699173872 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vbfsl" (UID: "3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5") : secret "cluster-monitoring-operator-tls" not found Aug 04 00:06:42.216612 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:42.216533 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 00:06:42.216612 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:42.216584 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls podName:e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb nodeName:}" failed. No retries permitted until 2026-08-04 00:06:50.216571523 +0000 UTC m=+256.699237830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-49dhp" (UID: "e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb") : secret "samples-operator-tls" not found Aug 04 00:06:42.318698 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.318670 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-bns6q"] Aug 04 00:06:42.319774 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.319758 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29763360-vng42_2f998954-55dc-4e58-a70a-ad878047d419/image-pruner/0.log" Aug 04 00:06:42.321740 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:06:42.321705 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b27d7a_dbe4_4d72_a8cf_71be2bea62c1.slice/crio-d6c942920ad87c2dd973996da0ee6601f8139fb740aab6956d500b115eda0734 WatchSource:0}: Error finding container d6c942920ad87c2dd973996da0ee6601f8139fb740aab6956d500b115eda0734: Status 404 returned error can't find the container with id d6c942920ad87c2dd973996da0ee6601f8139fb740aab6956d500b115eda0734 Aug 04 00:06:42.778402 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.778376 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-bns6q" event={"ID":"c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1","Type":"ContainerStarted","Data":"f3fc2aae7f54abb1e3c5955d8dd7b601b7d0db6e31b219b9f860e7b8a8e36a23"} Aug 04 00:06:42.778540 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.778419 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-bns6q" event={"ID":"c9b27d7a-dbe4-4d72-a8cf-71be2bea62c1","Type":"ContainerStarted","Data":"d6c942920ad87c2dd973996da0ee6601f8139fb740aab6956d500b115eda0734"} Aug 04 00:06:42.797138 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.797094 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-bns6q" podStartSLOduration=1.7970789109999998 podStartE2EDuration="1.797078911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:06:42.796041397 +0000 UTC m=+249.278707728" watchObservedRunningTime="2026-08-04 00:06:42.797078911 +0000 UTC m=+249.279745275" Aug 04 00:06:42.800204 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.800184 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:42.920174 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.920154 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f5cfv_db57e300-ed1e-4192-9977-6fac6d117f66/node-ca/0.log" Aug 04 00:06:42.922650 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.922629 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:42.922737 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:42.922691 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:42.922798 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:42.922738 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 00:06:42.922798 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:42.922776 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 00:06:42.922798 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:42.922790 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm podName:74a07772-f95e-453a-8694-1ccce1cf23ac nodeName:}" failed. No retries permitted until 2026-08-04 00:06:46.922777183 +0000 UTC m=+253.405443510 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cx6wf" (UID: "74a07772-f95e-453a-8694-1ccce1cf23ac") : configmap "kube-rbac-proxy" not found Aug 04 00:06:42.922959 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:42.922819 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls podName:74a07772-f95e-453a-8694-1ccce1cf23ac nodeName:}" failed. No retries permitted until 2026-08-04 00:06:46.92280614 +0000 UTC m=+253.405472456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cx6wf" (UID: "74a07772-f95e-453a-8694-1ccce1cf23ac") : secret "insights-runtime-extractor-tls" not found Aug 04 00:06:43.719217 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:43.719174 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fdgzw_df13c2f8-df07-47cb-82dc-2ac530b6985f/serve-healthcheck-canary/0.log" Aug 04 00:06:43.868146 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:43.868098 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:43.970551 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:43.970459 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:44.068367 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:44.068335 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:44.168988 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:44.168958 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:44.221454 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:44.221382 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:44.304615 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:44.304589 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:44.706302 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:44.706269 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:44.706471 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:44.706327 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:44.706664 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:44.706653 2570 scope.go:117] "RemoveContainer" containerID="ea5ff1a07b6bcc413a0f3d641fb02814d543f534217edf567d8f9e9d57f5add8" Aug 04 00:06:44.706818 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:44.706800 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-mhqk6_openshift-console-operator(036e01c1-b816-4ff5-8605-1ab0a192f4ee)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" podUID="036e01c1-b816-4ff5-8605-1ab0a192f4ee" Aug 04 00:06:44.771325 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:44.771294 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:45.363451 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:45.363391 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:45.707001 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:45.706916 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:45.745344 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:45.745310 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs\") pod \"network-metrics-daemon-9b87x\" (UID: \"e656f284-112c-4309-9151-41e5a9f45ede\") " pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:06:45.745501 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:45.745428 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 04 00:06:45.745501 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:45.745480 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs podName:e656f284-112c-4309-9151-41e5a9f45ede nodeName:}" failed. No retries permitted until 2026-08-04 00:08:47.745466587 +0000 UTC m=+374.228132896 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs") pod "network-metrics-daemon-9b87x" (UID: "e656f284-112c-4309-9151-41e5a9f45ede") : secret "metrics-daemon-secret" not found Aug 04 00:06:45.807682 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:45.807653 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:45.934024 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:45.933982 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:45.994904 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:45.994876 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:46.167212 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:46.167182 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:46.569096 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:46.569063 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:46.760315 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:46.760281 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:46.954443 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:46.954370 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:46.954595 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:46.954479 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:46.954595 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:46.954519 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 00:06:46.954595 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:46.954550 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 00:06:46.954595 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:46.954589 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls podName:74a07772-f95e-453a-8694-1ccce1cf23ac nodeName:}" failed. No retries permitted until 2026-08-04 00:06:54.954574404 +0000 UTC m=+261.437240716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cx6wf" (UID: "74a07772-f95e-453a-8694-1ccce1cf23ac") : secret "insights-runtime-extractor-tls" not found Aug 04 00:06:46.954731 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:46.954604 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm podName:74a07772-f95e-453a-8694-1ccce1cf23ac nodeName:}" failed. No retries permitted until 2026-08-04 00:06:54.954597249 +0000 UTC m=+261.437263557 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cx6wf" (UID: "74a07772-f95e-453a-8694-1ccce1cf23ac") : configmap "kube-rbac-proxy" not found Aug 04 00:06:47.167141 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:47.167103 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:47.763416 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:47.763388 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:47.814897 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:47.814870 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:47.891860 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:47.891830 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:48.161556 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:48.161483 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:48.367912 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:48.367866 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:48.961116 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:48.961082 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:49.210176 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:49.210146 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:49.282275 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:49.282247 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:49.768276 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:49.768246 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:50.160938 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:50.160866 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:50.280632 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:50.280599 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbfsl\" (UID: \"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" Aug 04 00:06:50.280796 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:50.280654 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-49dhp\" (UID: \"e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp" Aug 04 00:06:50.280796 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:50.280758 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 00:06:50.280877 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:06:50.280852 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls podName:3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5 nodeName:}" failed. No retries permitted until 2026-08-04 00:07:06.280832142 +0000 UTC m=+272.763498464 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vbfsl" (UID: "3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5") : secret "cluster-monitoring-operator-tls" not found Aug 04 00:06:50.283012 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:50.282985 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-49dhp\" (UID: \"e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp" Aug 04 00:06:50.300217 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:50.300192 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp" Aug 04 00:06:50.423297 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:50.423211 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp"] Aug 04 00:06:50.496122 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:50.496103 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:50.596909 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:50.596887 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:50.758806 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:50.758780 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:50.804119 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:50.804097 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp" event={"ID":"e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb","Type":"ContainerStarted","Data":"d455c240d6e2249b3abbbb966e5190045ab28023b98c1cf6d596f6dd14dcc68d"} Aug 04 00:06:51.012663 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:51.012617 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:51.166619 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:51.166588 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:51.969081 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:51.969046 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:52.164735 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:52.164686 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:52.565976 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:52.565944 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:52.758722 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:52.758698 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:52.812332 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:52.812303 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp" event={"ID":"e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb","Type":"ContainerStarted","Data":"866917a391172ad97f38a2a4067ef914bbc88bca95ac403b4052af33b2c3859c"} Aug 04 00:06:52.812431 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:52.812396 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp" event={"ID":"e68faaa2-16a6-46d0-8a1c-5d0ee1a4ecfb","Type":"ContainerStarted","Data":"b762ebdef01a70e71afe451c57854113512dbef6bb2850e270c084112a079a8f"} Aug 04 00:06:52.829125 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:52.829050 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-49dhp" podStartSLOduration=17.065953165 podStartE2EDuration="18.8290368s" totalImagesPullingTime="1.763083635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:06:52.828041442 +0000 UTC m=+259.310707773" watchObservedRunningTime="2026-08-04 00:06:52.8290368 +0000 UTC m=+259.311703159" Aug 04 00:06:53.161405 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:53.161355 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:53.846425 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:53.846399 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:53.962275 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:53.962253 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:54.165303 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:54.165243 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:54.564954 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:54.564923 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:55.015079 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:55.015046 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:55.015487 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:55.015099 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:55.015636 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:55.015603 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74a07772-f95e-453a-8694-1ccce1cf23ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:55.017675 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:55.017656 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74a07772-f95e-453a-8694-1ccce1cf23ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cx6wf\" (UID: \"74a07772-f95e-453a-8694-1ccce1cf23ac\") " pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:55.053062 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:55.053037 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cx6wf" Aug 04 00:06:55.178845 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:55.178818 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cx6wf"] Aug 04 00:06:55.181654 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:06:55.181629 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a07772_f95e_453a_8694_1ccce1cf23ac.slice/crio-807803e7f05fedbce72d2dc762494c8dcb77ddd187a677380b0bc5ddd9fe8214 WatchSource:0}: Error finding container 807803e7f05fedbce72d2dc762494c8dcb77ddd187a677380b0bc5ddd9fe8214: Status 404 returned error can't find the container with id 807803e7f05fedbce72d2dc762494c8dcb77ddd187a677380b0bc5ddd9fe8214 Aug 04 00:06:55.650419 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:55.650390 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:55.823713 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:55.823681 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cx6wf" event={"ID":"74a07772-f95e-453a-8694-1ccce1cf23ac","Type":"ContainerStarted","Data":"7f93ee3f581dbecf52604fea5c8751449491ed7d0508e4322e04775d87e1148b"} Aug 04 00:06:55.823847 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:55.823745 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cx6wf" event={"ID":"74a07772-f95e-453a-8694-1ccce1cf23ac","Type":"ContainerStarted","Data":"807803e7f05fedbce72d2dc762494c8dcb77ddd187a677380b0bc5ddd9fe8214"} Aug 04 00:06:56.028320 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:56.028290 2570 scope.go:117] "RemoveContainer" containerID="ea5ff1a07b6bcc413a0f3d641fb02814d543f534217edf567d8f9e9d57f5add8" Aug 04 00:06:56.833298 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:56.833258 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cx6wf" event={"ID":"74a07772-f95e-453a-8694-1ccce1cf23ac","Type":"ContainerStarted","Data":"df015f5a325d6adcef109624ea11f813ee1d6cc1b1366819be20649625c57e7b"} Aug 04 00:06:56.835738 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:56.835712 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:06:56.835885 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:56.835796 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" event={"ID":"036e01c1-b816-4ff5-8605-1ab0a192f4ee","Type":"ContainerStarted","Data":"b27e962edee883519cd57b0e68af0d133c5e42464ce211d4a086436a0b5ba3ed"} Aug 04 00:06:56.836240 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:56.836169 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:56.841947 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:56.841928 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" Aug 04 00:06:56.862968 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:56.862924 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-mhqk6" podStartSLOduration=19.773049885 podStartE2EDuration="22.862908592s" totalImagesPullingTime="3.089858707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:06:56.861509783 +0000 UTC m=+263.344176113" watchObservedRunningTime="2026-08-04 00:06:56.862908592 +0000 UTC m=+263.345574926" Aug 04 00:06:57.839948 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:57.839911 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:57.841065 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:57.841036 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cx6wf" event={"ID":"74a07772-f95e-453a-8694-1ccce1cf23ac","Type":"ContainerStarted","Data":"6d83bc78fbf460d08966066d828d1c401bdfb06c77bf9c09ba7dbb16c5ed07d1"} Aug 04 00:06:57.861812 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:57.861769 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cx6wf" podStartSLOduration=16.526443126 podStartE2EDuration="18.861754324s" totalImagesPullingTime="2.335311198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:06:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:06:57.859736037 +0000 UTC m=+264.342402369" watchObservedRunningTime="2026-08-04 00:06:57.861754324 +0000 UTC m=+264.344420654" Aug 04 00:06:57.965783 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:57.965730 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:58.019329 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:58.019306 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:58.392142 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:58.392118 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:58.493721 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:58.493697 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:58.593626 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:58.593601 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:58.693896 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:58.693838 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:06:58.794264 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:06:58.794239 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:03.546746 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:03.546718 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:03.647259 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:03.647218 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:03.754424 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:03.754390 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:06.297884 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:06.297841 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbfsl\" (UID: \"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" Aug 04 00:07:06.300141 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:06.300120 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbfsl\" (UID: \"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" Aug 04 00:07:06.490950 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:06.490915 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" Aug 04 00:07:06.627209 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:06.627181 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl"] Aug 04 00:07:06.630174 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:07:06.630146 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3021d0ff_dce6_45e6_a5f4_a0dae26dd3b5.slice/crio-e9b6b0ba5a58dabdd6a996de09f3746ce37f28b401752fde995527994faacf5a WatchSource:0}: Error finding container e9b6b0ba5a58dabdd6a996de09f3746ce37f28b401752fde995527994faacf5a: Status 404 returned error can't find the container with id e9b6b0ba5a58dabdd6a996de09f3746ce37f28b401752fde995527994faacf5a Aug 04 00:07:06.871805 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:06.871742 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" event={"ID":"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5","Type":"ContainerStarted","Data":"e9b6b0ba5a58dabdd6a996de09f3746ce37f28b401752fde995527994faacf5a"} Aug 04 00:07:07.603279 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:07.603243 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:07.703230 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:07.703189 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:08.297929 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:08.297899 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:08.404182 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:08.404141 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:08.796137 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:08.796102 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:08.879723 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:08.879694 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" event={"ID":"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5","Type":"ContainerStarted","Data":"7ade1b6ebb56d5d0d298c3ebebb234ddce2e674e9216eface87c16c6c05705d7"} Aug 04 00:07:08.896251 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:08.896194 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" podStartSLOduration=33.048014138 podStartE2EDuration="34.896177858s" totalImagesPullingTime="1.84816372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:08.895744134 +0000 UTC m=+275.378410465" watchObservedRunningTime="2026-08-04 00:07:08.896177858 +0000 UTC m=+275.378844191" Aug 04 00:07:08.896734 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:08.896718 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:08.955300 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:08.955275 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-5r7jx"] Aug 04 00:07:08.958672 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:08.958656 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-5r7jx" Aug 04 00:07:08.960651 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:08.960631 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Aug 04 00:07:08.961312 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:08.961286 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mpwsj\"" Aug 04 00:07:08.968681 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:08.968316 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:08.968811 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:08.968752 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-5r7jx"] Aug 04 00:07:09.018797 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:09.018778 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/48432f55-35f4-4e33-909e-da66939c6d6c-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-5r7jx\" (UID: \"48432f55-35f4-4e33-909e-da66939c6d6c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-5r7jx" Aug 04 00:07:09.119267 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:09.119206 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/48432f55-35f4-4e33-909e-da66939c6d6c-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-5r7jx\" (UID: \"48432f55-35f4-4e33-909e-da66939c6d6c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-5r7jx" Aug 04 00:07:09.121700 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:09.121679 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/48432f55-35f4-4e33-909e-da66939c6d6c-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-5r7jx\" (UID: \"48432f55-35f4-4e33-909e-da66939c6d6c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-5r7jx" Aug 04 00:07:09.124980 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:09.124961 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:09.275056 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:09.275037 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-5r7jx" Aug 04 00:07:09.394671 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:09.394617 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-5r7jx"] Aug 04 00:07:09.397568 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:07:09.397537 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48432f55_35f4_4e33_909e_da66939c6d6c.slice/crio-760c55b0f928290930a50066fa10a09b6bf351b8eb076c841db83c3ca6d045b6 WatchSource:0}: Error finding container 760c55b0f928290930a50066fa10a09b6bf351b8eb076c841db83c3ca6d045b6: Status 404 returned error can't find the container with id 760c55b0f928290930a50066fa10a09b6bf351b8eb076c841db83c3ca6d045b6 Aug 04 00:07:09.885233 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:09.885193 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-5r7jx" event={"ID":"48432f55-35f4-4e33-909e-da66939c6d6c","Type":"ContainerStarted","Data":"760c55b0f928290930a50066fa10a09b6bf351b8eb076c841db83c3ca6d045b6"} Aug 04 00:07:10.595018 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:10.594986 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:10.653992 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:10.653934 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:10.889966 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:10.889936 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-5r7jx" event={"ID":"48432f55-35f4-4e33-909e-da66939c6d6c","Type":"ContainerStarted","Data":"3fe2d111aa918765c91d5a8595e9b6281b7eacf009a17d840c6079f10df3b75f"} Aug 04 00:07:10.890399 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:10.890114 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-5r7jx" Aug 04 00:07:10.894409 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:10.894391 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-5r7jx" Aug 04 00:07:10.904889 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:10.904825 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-5r7jx" podStartSLOduration=1.9293758269999999 podStartE2EDuration="2.90481233s" totalImagesPullingTime="975.436503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:07:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:10.904171326 +0000 UTC m=+277.386837656" watchObservedRunningTime="2026-08-04 00:07:10.90481233 +0000 UTC m=+277.387478660" Aug 04 00:07:11.976495 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:11.976460 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:12.009720 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.009686 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-4s9hh"] Aug 04 00:07:12.043782 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.043756 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-4s9hh"] Aug 04 00:07:12.043940 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.043864 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" Aug 04 00:07:12.047409 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.047375 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Aug 04 00:07:12.047561 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.047384 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-p8j7x\"" Aug 04 00:07:12.047561 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.047433 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Aug 04 00:07:12.047561 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.047384 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 00:07:12.140705 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.140673 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/aef52237-4118-485e-8785-8e182bd64689-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-4s9hh\" (UID: \"aef52237-4118-485e-8785-8e182bd64689\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" Aug 04 00:07:12.140847 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.140799 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aef52237-4118-485e-8785-8e182bd64689-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-4s9hh\" (UID: \"aef52237-4118-485e-8785-8e182bd64689\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" Aug 04 00:07:12.140889 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.140859 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8448p\" (UniqueName: \"kubernetes.io/projected/aef52237-4118-485e-8785-8e182bd64689-kube-api-access-8448p\") pod \"prometheus-operator-758d7cb874-4s9hh\" (UID: \"aef52237-4118-485e-8785-8e182bd64689\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" Aug 04 00:07:12.140931 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.140884 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aef52237-4118-485e-8785-8e182bd64689-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-4s9hh\" (UID: \"aef52237-4118-485e-8785-8e182bd64689\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" Aug 04 00:07:12.143443 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.143423 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:12.241387 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.241360 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aef52237-4118-485e-8785-8e182bd64689-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-4s9hh\" (UID: \"aef52237-4118-485e-8785-8e182bd64689\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" Aug 04 00:07:12.241522 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.241396 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8448p\" (UniqueName: \"kubernetes.io/projected/aef52237-4118-485e-8785-8e182bd64689-kube-api-access-8448p\") pod \"prometheus-operator-758d7cb874-4s9hh\" (UID: \"aef52237-4118-485e-8785-8e182bd64689\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" Aug 04 00:07:12.241522 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.241413 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aef52237-4118-485e-8785-8e182bd64689-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-4s9hh\" (UID: \"aef52237-4118-485e-8785-8e182bd64689\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" Aug 04 00:07:12.241639 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.241587 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/aef52237-4118-485e-8785-8e182bd64689-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-4s9hh\" (UID: \"aef52237-4118-485e-8785-8e182bd64689\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" Aug 04 00:07:12.241752 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:07:12.241734 2570 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Aug 04 00:07:12.241828 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:07:12.241816 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aef52237-4118-485e-8785-8e182bd64689-prometheus-operator-tls podName:aef52237-4118-485e-8785-8e182bd64689 nodeName:}" failed. No retries permitted until 2026-08-04 00:07:12.741796402 +0000 UTC m=+279.224462712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/aef52237-4118-485e-8785-8e182bd64689-prometheus-operator-tls") pod "prometheus-operator-758d7cb874-4s9hh" (UID: "aef52237-4118-485e-8785-8e182bd64689") : secret "prometheus-operator-tls" not found Aug 04 00:07:12.242047 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.242021 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aef52237-4118-485e-8785-8e182bd64689-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-4s9hh\" (UID: \"aef52237-4118-485e-8785-8e182bd64689\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" Aug 04 00:07:12.243738 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.243711 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aef52237-4118-485e-8785-8e182bd64689-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-4s9hh\" (UID: \"aef52237-4118-485e-8785-8e182bd64689\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" Aug 04 00:07:12.249182 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.249160 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8448p\" (UniqueName: \"kubernetes.io/projected/aef52237-4118-485e-8785-8e182bd64689-kube-api-access-8448p\") pod \"prometheus-operator-758d7cb874-4s9hh\" (UID: \"aef52237-4118-485e-8785-8e182bd64689\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" Aug 04 00:07:12.295124 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.295104 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:12.745643 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.745615 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/aef52237-4118-485e-8785-8e182bd64689-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-4s9hh\" (UID: \"aef52237-4118-485e-8785-8e182bd64689\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" Aug 04 00:07:12.747925 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.747907 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/aef52237-4118-485e-8785-8e182bd64689-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-4s9hh\" (UID: \"aef52237-4118-485e-8785-8e182bd64689\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" Aug 04 00:07:12.953997 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.953964 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" Aug 04 00:07:12.983005 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:12.982977 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:13.039529 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:13.039496 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:13.097718 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:13.097627 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-4s9hh"] Aug 04 00:07:13.100136 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:07:13.100107 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef52237_4118_485e_8785_8e182bd64689.slice/crio-c733084e80cbaed1337af40cb7d0217810a776fe3d1dc8ebfe2aa975a68449f3 WatchSource:0}: Error finding container c733084e80cbaed1337af40cb7d0217810a776fe3d1dc8ebfe2aa975a68449f3: Status 404 returned error can't find the container with id c733084e80cbaed1337af40cb7d0217810a776fe3d1dc8ebfe2aa975a68449f3 Aug 04 00:07:13.905502 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:13.905461 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" event={"ID":"aef52237-4118-485e-8785-8e182bd64689","Type":"ContainerStarted","Data":"c733084e80cbaed1337af40cb7d0217810a776fe3d1dc8ebfe2aa975a68449f3"} Aug 04 00:07:14.619306 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:14.619164 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:14.910749 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:14.910656 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" event={"ID":"aef52237-4118-485e-8785-8e182bd64689","Type":"ContainerStarted","Data":"91ba1fe97ed8c317f121c287bad58455d2ce811b2bf74d077a6d7c21f873b867"} Aug 04 00:07:14.910749 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:14.910711 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" event={"ID":"aef52237-4118-485e-8785-8e182bd64689","Type":"ContainerStarted","Data":"42c27e5f0fd594ab7f721d7d087e2f5ad5c1b4e40a5d00225e4fb90f70ae5026"} Aug 04 00:07:14.926881 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:14.926828 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-4s9hh" podStartSLOduration=2.669893224 podStartE2EDuration="3.926814884s" totalImagesPullingTime="1.25692166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:14.925846199 +0000 UTC m=+281.408512530" watchObservedRunningTime="2026-08-04 00:07:14.926814884 +0000 UTC m=+281.409481213" Aug 04 00:07:15.522134 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:07:15.522080 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" podUID="7a3fc7b1-2b51-4009-ade1-d8b2c609a20c" Aug 04 00:07:15.522134 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:07:15.522080 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-88l8k" podUID="c3ee0892-91f7-4cab-8b96-829a660ab62d" Aug 04 00:07:15.913553 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:15.913473 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:07:15.913553 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:15.913487 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-88l8k" Aug 04 00:07:16.110263 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:16.110215 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:16.322364 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:16.322334 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:16.918195 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:16.918167 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:07:16.918579 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:16.918205 2570 generic.go:356] "Generic (PLEG): container finished" podID="3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5" containerID="7ade1b6ebb56d5d0d298c3ebebb234ddce2e674e9216eface87c16c6c05705d7" exitCode=2 Aug 04 00:07:16.918579 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:16.918285 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" event={"ID":"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5","Type":"ContainerDied","Data":"7ade1b6ebb56d5d0d298c3ebebb234ddce2e674e9216eface87c16c6c05705d7"} Aug 04 00:07:16.918666 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:16.918578 2570 scope.go:117] "RemoveContainer" containerID="7ade1b6ebb56d5d0d298c3ebebb234ddce2e674e9216eface87c16c6c05705d7" Aug 04 00:07:17.927516 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:17.927488 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:07:17.927920 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:17.927580 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbfsl" event={"ID":"3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5","Type":"ContainerStarted","Data":"eb57644512e3f80c97c5e57078af78fa58530ab5de5da833cb17dba21994c228"} Aug 04 00:07:18.619596 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:18.619562 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:18.678555 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:18.678528 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:18.792008 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:18.791965 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:07:18.794389 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:18.794361 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls\") pod \"image-registry-7fc48d6bdf-m8vf4\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:07:18.893142 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:18.893055 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:07:18.895390 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:18.895366 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3ee0892-91f7-4cab-8b96-829a660ab62d-metrics-tls\") pod \"dns-default-88l8k\" (UID: \"c3ee0892-91f7-4cab-8b96-829a660ab62d\") " pod="openshift-dns/dns-default-88l8k" Aug 04 00:07:18.917151 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:18.917126 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rbxcz\"" Aug 04 00:07:18.918108 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:18.918094 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7ds4t\"" Aug 04 00:07:18.924924 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:18.924908 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:07:18.925020 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:18.925003 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-88l8k" Aug 04 00:07:19.082846 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:19.082816 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fc48d6bdf-m8vf4"] Aug 04 00:07:19.085855 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:07:19.085830 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3fc7b1_2b51_4009_ade1_d8b2c609a20c.slice/crio-eb6f6f161482695c5bf9d3a5a8d48a48d004556fa3c81056bbed842e21f2caea WatchSource:0}: Error finding container eb6f6f161482695c5bf9d3a5a8d48a48d004556fa3c81056bbed842e21f2caea: Status 404 returned error can't find the container with id eb6f6f161482695c5bf9d3a5a8d48a48d004556fa3c81056bbed842e21f2caea Aug 04 00:07:19.086187 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:19.086168 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-88l8k"] Aug 04 00:07:19.088079 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:07:19.088060 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ee0892_91f7_4cab_8b96_829a660ab62d.slice/crio-34e7591e5ac1d48da2723320f1bd2dafcb3731055c7bc0e0126f41525f9ea9da WatchSource:0}: Error finding container 34e7591e5ac1d48da2723320f1bd2dafcb3731055c7bc0e0126f41525f9ea9da: Status 404 returned error can't find the container with id 34e7591e5ac1d48da2723320f1bd2dafcb3731055c7bc0e0126f41525f9ea9da Aug 04 00:07:19.936678 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:19.936640 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" event={"ID":"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c","Type":"ContainerStarted","Data":"ce54e2f3ed792acb694ee57afd9b40c83601ea10258f7bb543e1fb39a1a8d751"} Aug 04 00:07:19.936873 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:19.936699 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" event={"ID":"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c","Type":"ContainerStarted","Data":"eb6f6f161482695c5bf9d3a5a8d48a48d004556fa3c81056bbed842e21f2caea"} Aug 04 00:07:19.936873 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:19.936749 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:07:19.938002 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:19.937975 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-88l8k" event={"ID":"c3ee0892-91f7-4cab-8b96-829a660ab62d","Type":"ContainerStarted","Data":"34e7591e5ac1d48da2723320f1bd2dafcb3731055c7bc0e0126f41525f9ea9da"} Aug 04 00:07:19.958803 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:19.958761 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" podStartSLOduration=285.958748865 podStartE2EDuration="4m45.958748865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:19.956420014 +0000 UTC m=+286.439086357" watchObservedRunningTime="2026-08-04 00:07:19.958748865 +0000 UTC m=+286.441415174" Aug 04 00:07:20.543343 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.543271 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:20.595422 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.595392 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-svq5p"] Aug 04 00:07:20.598002 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.597983 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.601045 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.600518 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 00:07:20.601045 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.600715 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 00:07:20.601045 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.600922 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6k5hv\"" Aug 04 00:07:20.601264 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.601069 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 00:07:20.642920 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.642849 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:20.708586 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.708563 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-textfile\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.708716 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.708611 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-accelerators-collector-config\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.708716 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.708649 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/251d8954-f1bf-4a73-822b-784c13856b22-sys\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.708716 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.708674 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-wtmp\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.708884 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.708719 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8fr6\" (UniqueName: \"kubernetes.io/projected/251d8954-f1bf-4a73-822b-784c13856b22-kube-api-access-j8fr6\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.708884 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.708748 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-tls\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.708884 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.708799 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/251d8954-f1bf-4a73-822b-784c13856b22-metrics-client-ca\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.708884 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.708844 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/251d8954-f1bf-4a73-822b-784c13856b22-root\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.709060 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.708892 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.735543 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.735507 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:20.810198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.810120 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-wtmp\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.810198 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.810181 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8fr6\" (UniqueName: \"kubernetes.io/projected/251d8954-f1bf-4a73-822b-784c13856b22-kube-api-access-j8fr6\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.810402 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.810206 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-tls\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.810402 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.810277 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/251d8954-f1bf-4a73-822b-784c13856b22-metrics-client-ca\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.810402 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.810303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/251d8954-f1bf-4a73-822b-784c13856b22-root\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.810402 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.810340 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.810402 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.810343 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-wtmp\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.810402 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.810370 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-textfile\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.810702 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.810427 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-accelerators-collector-config\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.810702 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.810465 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/251d8954-f1bf-4a73-822b-784c13856b22-sys\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.810702 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.810531 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/251d8954-f1bf-4a73-822b-784c13856b22-sys\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.810702 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.810570 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/251d8954-f1bf-4a73-822b-784c13856b22-root\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.810895 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:07:20.810832 2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Aug 04 00:07:20.810895 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:07:20.810884 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-tls podName:251d8954-f1bf-4a73-822b-784c13856b22 nodeName:}" failed. No retries permitted until 2026-08-04 00:07:21.310867627 +0000 UTC m=+287.793533950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-tls") pod "node-exporter-svq5p" (UID: "251d8954-f1bf-4a73-822b-784c13856b22") : secret "node-exporter-tls" not found Aug 04 00:07:20.811002 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.810979 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/251d8954-f1bf-4a73-822b-784c13856b22-metrics-client-ca\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.811053 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.811008 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-accelerators-collector-config\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.811292 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.811258 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-textfile\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.813860 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.813705 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.819917 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.819878 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8fr6\" (UniqueName: \"kubernetes.io/projected/251d8954-f1bf-4a73-822b-784c13856b22-kube-api-access-j8fr6\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:20.836843 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.836817 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:20.936638 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.936614 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:20.944111 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:20.944091 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-88l8k" event={"ID":"c3ee0892-91f7-4cab-8b96-829a660ab62d","Type":"ContainerStarted","Data":"e15a651d135c31ec358bf39056b4f9afe4a546680c863060011df3f0014c7259"} Aug 04 00:07:21.037278 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.037217 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:21.138553 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.138482 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:21.255021 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.254992 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:21.313941 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.313915 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-tls\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:21.316321 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.316298 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/251d8954-f1bf-4a73-822b-784c13856b22-node-exporter-tls\") pod \"node-exporter-svq5p\" (UID: \"251d8954-f1bf-4a73-822b-784c13856b22\") " pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:21.511338 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.511312 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-svq5p" Aug 04 00:07:21.520677 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:07:21.520656 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251d8954_f1bf_4a73_822b_784c13856b22.slice/crio-bc325dcce66d4b1ad6d7ed0a835cc8509ac18dbb03e63868c48978fc7f5ef3d4 WatchSource:0}: Error finding container bc325dcce66d4b1ad6d7ed0a835cc8509ac18dbb03e63868c48978fc7f5ef3d4: Status 404 returned error can't find the container with id bc325dcce66d4b1ad6d7ed0a835cc8509ac18dbb03e63868c48978fc7f5ef3d4 Aug 04 00:07:21.655937 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.655913 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:21.739310 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.739289 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 00:07:21.741142 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.741115 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:21.742524 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.742506 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.745607 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.745586 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 04 00:07:21.745607 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.745600 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 04 00:07:21.745810 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.745629 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 04 00:07:21.745810 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.745629 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 04 00:07:21.746028 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.745971 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 04 00:07:21.746028 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.745989 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 04 00:07:21.746028 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.746007 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 04 00:07:21.746175 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.746051 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 04 00:07:21.746319 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.746304 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gspz6\"" Aug 04 00:07:21.747162 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.747145 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 04 00:07:21.759029 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.759009 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 00:07:21.817898 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.817840 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2621b7bb-8947-4bf3-af30-4817c5731b3f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.817898 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.817870 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.817898 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.817889 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2621b7bb-8947-4bf3-af30-4817c5731b3f-config-out\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.818111 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.817906 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.818111 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.817970 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2621b7bb-8947-4bf3-af30-4817c5731b3f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.818111 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.817996 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.818111 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.818021 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjfps\" (UniqueName: \"kubernetes.io/projected/2621b7bb-8947-4bf3-af30-4817c5731b3f-kube-api-access-vjfps\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.818111 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.818047 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2621b7bb-8947-4bf3-af30-4817c5731b3f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.818111 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.818067 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2621b7bb-8947-4bf3-af30-4817c5731b3f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.818111 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.818107 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.818485 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.818123 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-web-config\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.818485 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.818140 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-config-volume\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.818485 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.818157 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.863576 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.863547 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:21.919079 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.919056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2621b7bb-8947-4bf3-af30-4817c5731b3f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.919176 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.919086 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2621b7bb-8947-4bf3-af30-4817c5731b3f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.919176 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.919114 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.919176 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.919137 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-web-config\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.919176 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.919165 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-config-volume\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.919407 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.919189 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.919407 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.919356 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2621b7bb-8947-4bf3-af30-4817c5731b3f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.919407 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.919389 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.919566 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.919416 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2621b7bb-8947-4bf3-af30-4817c5731b3f-config-out\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.919566 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.919444 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.919566 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.919483 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2621b7bb-8947-4bf3-af30-4817c5731b3f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.919566 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.919523 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.919763 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.919563 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjfps\" (UniqueName: \"kubernetes.io/projected/2621b7bb-8947-4bf3-af30-4817c5731b3f-kube-api-access-vjfps\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.920596 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.920278 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2621b7bb-8947-4bf3-af30-4817c5731b3f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.920596 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.920401 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2621b7bb-8947-4bf3-af30-4817c5731b3f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.920596 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.920545 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2621b7bb-8947-4bf3-af30-4817c5731b3f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.922438 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.922414 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-config-volume\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.922547 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.922448 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.922792 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.922753 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2621b7bb-8947-4bf3-af30-4817c5731b3f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.923209 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.923185 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2621b7bb-8947-4bf3-af30-4817c5731b3f-config-out\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.923319 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.923281 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.923319 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.923303 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.923824 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.923804 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.923957 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.923939 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-web-config\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.924943 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.924924 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2621b7bb-8947-4bf3-af30-4817c5731b3f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.927785 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.927764 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjfps\" (UniqueName: \"kubernetes.io/projected/2621b7bb-8947-4bf3-af30-4817c5731b3f-kube-api-access-vjfps\") pod \"alertmanager-main-0\" (UID: \"2621b7bb-8947-4bf3-af30-4817c5731b3f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:21.949418 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.949395 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-svq5p" event={"ID":"251d8954-f1bf-4a73-822b-784c13856b22","Type":"ContainerStarted","Data":"bc325dcce66d4b1ad6d7ed0a835cc8509ac18dbb03e63868c48978fc7f5ef3d4"} Aug 04 00:07:21.951065 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.951037 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-88l8k" event={"ID":"c3ee0892-91f7-4cab-8b96-829a660ab62d","Type":"ContainerStarted","Data":"42f23d302c1457ef69ba40a63d328d1541ff8b9b0980199dc89dc3a427ce4ffb"} Aug 04 00:07:21.951183 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.951168 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-88l8k" Aug 04 00:07:21.968684 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:21.968640 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-88l8k" podStartSLOduration=252.202597625 podStartE2EDuration="4m13.968628808s" totalImagesPullingTime="1.766031183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:21.967881112 +0000 UTC m=+288.450547459" watchObservedRunningTime="2026-08-04 00:07:21.968628808 +0000 UTC m=+288.451295138" Aug 04 00:07:22.053869 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:22.053847 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 00:07:22.120486 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:22.120452 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:22.179941 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:22.179877 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:22.212922 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:22.212892 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 00:07:22.216956 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:07:22.216932 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2621b7bb_8947_4bf3_af30_4817c5731b3f.slice/crio-2bce858d4ca5ea173c4551149e4804130424aab9464cc944ffbf3bfbb03fa5c0 WatchSource:0}: Error finding container 2bce858d4ca5ea173c4551149e4804130424aab9464cc944ffbf3bfbb03fa5c0: Status 404 returned error can't find the container with id 2bce858d4ca5ea173c4551149e4804130424aab9464cc944ffbf3bfbb03fa5c0 Aug 04 00:07:22.957088 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:22.957009 2570 generic.go:356] "Generic (PLEG): container finished" podID="251d8954-f1bf-4a73-822b-784c13856b22" containerID="11fad9cb84c98373946c00991ec42cd1c130d4d0f7fa946c463783238d2c63af" exitCode=0 Aug 04 00:07:22.957466 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:22.957101 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-svq5p" event={"ID":"251d8954-f1bf-4a73-822b-784c13856b22","Type":"ContainerDied","Data":"11fad9cb84c98373946c00991ec42cd1c130d4d0f7fa946c463783238d2c63af"} Aug 04 00:07:22.958679 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:22.958654 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2621b7bb-8947-4bf3-af30-4817c5731b3f","Type":"ContainerStarted","Data":"2bce858d4ca5ea173c4551149e4804130424aab9464cc944ffbf3bfbb03fa5c0"} Aug 04 00:07:23.555345 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:23.555313 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:23.642038 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:23.641998 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:23.704062 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:23.704034 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:23.763983 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:23.763958 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:23.965791 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:23.965762 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-svq5p" event={"ID":"251d8954-f1bf-4a73-822b-784c13856b22","Type":"ContainerStarted","Data":"cbc54160188edbe70111a8e7a0cb9c486d617a28f389991db48fc8de4a006ff9"} Aug 04 00:07:23.966164 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:23.965821 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-svq5p" event={"ID":"251d8954-f1bf-4a73-822b-784c13856b22","Type":"ContainerStarted","Data":"8e70e0b70c4ccf9f596c445cc717538ace7518129b61d735b226ab785e1b0bbf"} Aug 04 00:07:23.967261 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:23.967217 2570 generic.go:356] "Generic (PLEG): container finished" podID="2621b7bb-8947-4bf3-af30-4817c5731b3f" containerID="939d253cd5f23a1a63fab51e5c3019a8ebaec8c3f64f97afbe7be6816088b708" exitCode=0 Aug 04 00:07:23.967365 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:23.967285 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2621b7bb-8947-4bf3-af30-4817c5731b3f","Type":"ContainerDied","Data":"939d253cd5f23a1a63fab51e5c3019a8ebaec8c3f64f97afbe7be6816088b708"} Aug 04 00:07:23.987938 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:23.987893 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-svq5p" podStartSLOduration=2.917793364 podStartE2EDuration="3.987880242s" totalImagesPullingTime="1.007612416s" totalInitContainerRuntime="62.474462ms" isStatefulPod=true podCreationTimestamp="2026-08-04 00:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:23.987027849 +0000 UTC m=+290.469694179" watchObservedRunningTime="2026-08-04 00:07:23.987880242 +0000 UTC m=+290.470546571" Aug 04 00:07:25.405527 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.403670 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:25.499938 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.499900 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:25.689920 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.689874 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:25.805557 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.805496 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w"] Aug 04 00:07:25.808521 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.808506 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:25.810907 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.810885 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Aug 04 00:07:25.811005 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.810936 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Aug 04 00:07:25.811257 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.811219 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Aug 04 00:07:25.811257 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.811250 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Aug 04 00:07:25.811392 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.811236 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Aug 04 00:07:25.811392 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.811328 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2h6wt\"" Aug 04 00:07:25.815658 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.815636 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Aug 04 00:07:25.820285 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.820267 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w"] Aug 04 00:07:25.953155 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.953130 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e22a206c-3881-4cff-8428-850d58b7b51d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:25.953272 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.953161 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e22a206c-3881-4cff-8428-850d58b7b51d-secret-telemeter-client\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:25.953272 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.953192 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e22a206c-3881-4cff-8428-850d58b7b51d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:25.953272 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.953270 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e22a206c-3881-4cff-8428-850d58b7b51d-serving-certs-ca-bundle\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:25.953379 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.953295 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e22a206c-3881-4cff-8428-850d58b7b51d-telemeter-client-tls\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:25.953379 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.953360 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e22a206c-3881-4cff-8428-850d58b7b51d-federate-client-tls\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:25.953444 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.953413 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pb9f\" (UniqueName: \"kubernetes.io/projected/e22a206c-3881-4cff-8428-850d58b7b51d-kube-api-access-5pb9f\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:25.953475 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.953437 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e22a206c-3881-4cff-8428-850d58b7b51d-metrics-client-ca\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:25.978679 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.978654 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2621b7bb-8947-4bf3-af30-4817c5731b3f","Type":"ContainerStarted","Data":"e187e08242fd54abd694707508c77cd203cac14225f7ca3f5c86e7feba600a1f"} Aug 04 00:07:25.978768 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.978708 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2621b7bb-8947-4bf3-af30-4817c5731b3f","Type":"ContainerStarted","Data":"1431288ea195076b5ea60386c4ddef1e89d6124273b8f4270a9e306b691aad05"} Aug 04 00:07:25.978768 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.978736 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2621b7bb-8947-4bf3-af30-4817c5731b3f","Type":"ContainerStarted","Data":"d99b6a552a2a40cd5958de514c2d1639c5141c184f9087f0528ee3755207fb56"} Aug 04 00:07:25.978768 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.978759 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2621b7bb-8947-4bf3-af30-4817c5731b3f","Type":"ContainerStarted","Data":"a9c0926283bf6c3fd5fcf5b3ad0418071fa1e3d5f2e86ea916239c5949db7d25"} Aug 04 00:07:25.978865 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:25.978784 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2621b7bb-8947-4bf3-af30-4817c5731b3f","Type":"ContainerStarted","Data":"200dbb5cc53d0a74d351b070c630ff589a5be9e3650d945c9eefc22677b9a776"} Aug 04 00:07:26.029588 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:07:26.029558 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9b87x" podUID="e656f284-112c-4309-9151-41e5a9f45ede" Aug 04 00:07:26.054323 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.054299 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e22a206c-3881-4cff-8428-850d58b7b51d-secret-telemeter-client\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:26.054487 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.054345 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e22a206c-3881-4cff-8428-850d58b7b51d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:26.054487 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.054394 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e22a206c-3881-4cff-8428-850d58b7b51d-serving-certs-ca-bundle\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:26.054487 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.054429 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e22a206c-3881-4cff-8428-850d58b7b51d-telemeter-client-tls\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:26.054645 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.054495 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e22a206c-3881-4cff-8428-850d58b7b51d-federate-client-tls\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:26.054645 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.054550 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pb9f\" (UniqueName: \"kubernetes.io/projected/e22a206c-3881-4cff-8428-850d58b7b51d-kube-api-access-5pb9f\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:26.054645 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.054585 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e22a206c-3881-4cff-8428-850d58b7b51d-metrics-client-ca\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:26.054645 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.054624 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e22a206c-3881-4cff-8428-850d58b7b51d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:26.055205 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.055179 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e22a206c-3881-4cff-8428-850d58b7b51d-serving-certs-ca-bundle\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:26.055596 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.055543 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e22a206c-3881-4cff-8428-850d58b7b51d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:26.055852 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.055825 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e22a206c-3881-4cff-8428-850d58b7b51d-metrics-client-ca\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:26.057261 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.057208 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e22a206c-3881-4cff-8428-850d58b7b51d-telemeter-client-tls\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:26.057457 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.057425 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e22a206c-3881-4cff-8428-850d58b7b51d-federate-client-tls\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:26.057606 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.057589 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e22a206c-3881-4cff-8428-850d58b7b51d-secret-telemeter-client\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:26.057799 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.057779 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e22a206c-3881-4cff-8428-850d58b7b51d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:26.062864 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.062839 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pb9f\" (UniqueName: \"kubernetes.io/projected/e22a206c-3881-4cff-8428-850d58b7b51d-kube-api-access-5pb9f\") pod \"telemeter-client-5674c5f7bf-hdb6w\" (UID: \"e22a206c-3881-4cff-8428-850d58b7b51d\") " pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:26.118810 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.118789 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" Aug 04 00:07:26.253712 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.253678 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w"] Aug 04 00:07:26.296951 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:07:26.296925 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode22a206c_3881_4cff_8428_850d58b7b51d.slice/crio-70661ad8436575bd63d75cc4064b6532a47a64658d833a4076b37226afccd494 WatchSource:0}: Error finding container 70661ad8436575bd63d75cc4064b6532a47a64658d833a4076b37226afccd494: Status 404 returned error can't find the container with id 70661ad8436575bd63d75cc4064b6532a47a64658d833a4076b37226afccd494 Aug 04 00:07:26.797885 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.797842 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:26.806009 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.805981 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 00:07:26.810742 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.810717 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.813519 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.813387 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Aug 04 00:07:26.813519 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.813405 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Aug 04 00:07:26.813519 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.813491 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Aug 04 00:07:26.817493 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.817464 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Aug 04 00:07:26.818525 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.817769 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Aug 04 00:07:26.818525 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.818217 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Aug 04 00:07:26.819356 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.818982 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Aug 04 00:07:26.819763 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.819704 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Aug 04 00:07:26.820138 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.820113 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Aug 04 00:07:26.820578 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.820559 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Aug 04 00:07:26.821030 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.821012 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Aug 04 00:07:26.821315 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.821299 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Aug 04 00:07:26.821647 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.821627 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4kpmms6c0i6i\"" Aug 04 00:07:26.826821 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.826680 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Aug 04 00:07:26.827014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.826994 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-q9t4h\"" Aug 04 00:07:26.827298 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.827281 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Aug 04 00:07:26.828815 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.828796 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Aug 04 00:07:26.830916 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.830883 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 00:07:26.961876 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.961832 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.961876 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.961868 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962081 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.961911 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962081 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.961937 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962081 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.961965 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0245384-7186-41f9-a102-a48b908f6735-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962081 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.962011 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962081 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.962044 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962081 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.962068 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962400 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.962095 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962400 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.962116 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962400 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.962137 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0245384-7186-41f9-a102-a48b908f6735-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962400 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.962154 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962400 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.962193 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962400 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.962261 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962400 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.962322 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0245384-7186-41f9-a102-a48b908f6735-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962400 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.962348 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhkdc\" (UniqueName: \"kubernetes.io/projected/c0245384-7186-41f9-a102-a48b908f6735-kube-api-access-rhkdc\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962400 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.962397 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962795 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.962425 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962795 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.962456 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.962795 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.962479 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-config\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:26.990168 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.990044 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2621b7bb-8947-4bf3-af30-4817c5731b3f","Type":"ContainerStarted","Data":"678538e1824fc71d46614c159c08dc7c24c3116051c00002936530b24c768c87"} Aug 04 00:07:26.992597 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:26.992568 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" event={"ID":"e22a206c-3881-4cff-8428-850d58b7b51d","Type":"ContainerStarted","Data":"70661ad8436575bd63d75cc4064b6532a47a64658d833a4076b37226afccd494"} Aug 04 00:07:27.064160 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064080 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0245384-7186-41f9-a102-a48b908f6735-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064160 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064116 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhkdc\" (UniqueName: \"kubernetes.io/projected/c0245384-7186-41f9-a102-a48b908f6735-kube-api-access-rhkdc\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064366 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064181 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064366 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064206 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064366 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064249 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064366 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064272 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-config\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064564 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064379 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064564 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064409 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064564 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064477 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064564 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064501 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064564 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064528 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0245384-7186-41f9-a102-a48b908f6735-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064789 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064574 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064789 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064603 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064789 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064627 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064789 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064668 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064789 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064705 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064789 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064736 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0245384-7186-41f9-a102-a48b908f6735-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064789 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064762 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.064789 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064787 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.065146 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.064811 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.066712 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.065721 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0245384-7186-41f9-a102-a48b908f6735-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.066712 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.066298 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.066712 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.066406 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.066712 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.066702 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.067679 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.067238 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.067679 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.067577 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.071121 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.069667 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0245384-7186-41f9-a102-a48b908f6735-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.071121 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.069835 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0245384-7186-41f9-a102-a48b908f6735-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.071121 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.070289 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.071121 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.070571 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.071121 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.070988 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.071121 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.071049 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.071121 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.071064 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0245384-7186-41f9-a102-a48b908f6735-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.072032 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.071977 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.073763 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.073714 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.074442 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.074302 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.074442 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.074405 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.074581 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.074461 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-config\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.075724 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.075698 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0245384-7186-41f9-a102-a48b908f6735-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.077896 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.077876 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhkdc\" (UniqueName: \"kubernetes.io/projected/c0245384-7186-41f9-a102-a48b908f6735-kube-api-access-rhkdc\") pod \"prometheus-k8s-0\" (UID: \"c0245384-7186-41f9-a102-a48b908f6735\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.132983 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.132955 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:27.310480 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.310408 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.171317478 podStartE2EDuration="6.310383298s" totalImagesPullingTime="3.121331349s" totalInitContainerRuntime="17.734471ms" isStatefulPod=false podCreationTimestamp="2026-08-04 00:07:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:27.033960022 +0000 UTC m=+293.516626353" watchObservedRunningTime="2026-08-04 00:07:27.310383298 +0000 UTC m=+293.793049630" Aug 04 00:07:27.313201 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.313170 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 00:07:27.318381 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:07:27.318355 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0245384_7186_41f9_a102_a48b908f6735.slice/crio-b319e242857fe768d293325f94e7ee080a5f7193eb4142f7b7222d4ed0f19630 WatchSource:0}: Error finding container b319e242857fe768d293325f94e7ee080a5f7193eb4142f7b7222d4ed0f19630: Status 404 returned error can't find the container with id b319e242857fe768d293325f94e7ee080a5f7193eb4142f7b7222d4ed0f19630 Aug 04 00:07:27.997762 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.997736 2570 generic.go:356] "Generic (PLEG): container finished" podID="c0245384-7186-41f9-a102-a48b908f6735" containerID="dd52c5ae9fcc9b4a33e7559d20efb4c8d17c5f43ac7952be15057986363a727e" exitCode=0 Aug 04 00:07:27.998094 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.997838 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0245384-7186-41f9-a102-a48b908f6735","Type":"ContainerDied","Data":"dd52c5ae9fcc9b4a33e7559d20efb4c8d17c5f43ac7952be15057986363a727e"} Aug 04 00:07:27.998094 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:27.997930 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0245384-7186-41f9-a102-a48b908f6735","Type":"ContainerStarted","Data":"b319e242857fe768d293325f94e7ee080a5f7193eb4142f7b7222d4ed0f19630"} Aug 04 00:07:29.005237 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:29.005190 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" event={"ID":"e22a206c-3881-4cff-8428-850d58b7b51d","Type":"ContainerStarted","Data":"2fb677a81def78a1eb1b39d0ec8003c792328644f598694f505e42dc716ec7a0"} Aug 04 00:07:29.005621 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:29.005310 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" event={"ID":"e22a206c-3881-4cff-8428-850d58b7b51d","Type":"ContainerStarted","Data":"b6ade57d32aa897fff366bea50fdd018035c84ec31fff5040dd08855e4da1082"} Aug 04 00:07:29.005621 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:29.005347 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" event={"ID":"e22a206c-3881-4cff-8428-850d58b7b51d","Type":"ContainerStarted","Data":"c64931aebcbd52cbab16403ce5ae709f3296f77fdb4e6ae1bdd8c6fce6a8314d"} Aug 04 00:07:29.028916 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:29.028871 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5674c5f7bf-hdb6w" podStartSLOduration=2.352954233 podStartE2EDuration="4.028857616s" totalImagesPullingTime="1.675903383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:29.025800185 +0000 UTC m=+295.508466515" watchObservedRunningTime="2026-08-04 00:07:29.028857616 +0000 UTC m=+295.511523946" Aug 04 00:07:29.400460 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:29.400384 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:29.458109 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:29.458082 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:31.960523 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:31.960491 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-88l8k" Aug 04 00:07:32.024085 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:32.024046 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0245384-7186-41f9-a102-a48b908f6735","Type":"ContainerStarted","Data":"00469afdaebec4d9f75bad612083283bc0d9a8f16d74dfa5e47564f7c0f532f0"} Aug 04 00:07:32.024402 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:32.024381 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0245384-7186-41f9-a102-a48b908f6735","Type":"ContainerStarted","Data":"fd0677659a61b87fdd4077c5a1abdddeac55f08484978ce7b260010bde749a2f"} Aug 04 00:07:33.921408 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:33.921373 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:07:33.922656 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:33.922637 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:07:33.923935 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:33.923916 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:07:33.925190 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:33.925172 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:07:33.929829 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:33.929806 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:07:33.931143 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:33.931125 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:07:33.935835 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:33.935817 2570 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 00:07:34.040100 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:34.039998 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0245384-7186-41f9-a102-a48b908f6735","Type":"ContainerStarted","Data":"669d9de0df4624bac3994c2c861230446c2ffe411d44c3f29148b8452c1d1ebd"} Aug 04 00:07:34.040100 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:34.040054 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0245384-7186-41f9-a102-a48b908f6735","Type":"ContainerStarted","Data":"65864a1a7e38bfdfcf8d6c3be79c5f9f83c92186e1ff032c17abb86b1556c269"} Aug 04 00:07:34.040100 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:34.040079 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0245384-7186-41f9-a102-a48b908f6735","Type":"ContainerStarted","Data":"16786ac875e19c9b77dafcaef64f38c742925c96155b7c64d28b517e94cc3943"} Aug 04 00:07:34.050004 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:34.040103 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0245384-7186-41f9-a102-a48b908f6735","Type":"ContainerStarted","Data":"98abef51d91683db58061ac62b66f73679e26157e4c55d0a0e0aa94a791bb789"} Aug 04 00:07:34.068385 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:34.068338 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.769669934 podStartE2EDuration="8.068320124s" totalImagesPullingTime="5.274099563s" totalInitContainerRuntime="24.550627ms" isStatefulPod=false podCreationTimestamp="2026-08-04 00:07:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:34.06608856 +0000 UTC m=+300.548754889" watchObservedRunningTime="2026-08-04 00:07:34.068320124 +0000 UTC m=+300.550986456" Aug 04 00:07:35.585667 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:35.585634 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:35.686204 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:35.686174 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:35.836917 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:35.836844 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fc48d6bdf-m8vf4"] Aug 04 00:07:35.840958 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:35.840926 2570 patch_prober.go:28] interesting pod/image-registry-7fc48d6bdf-m8vf4 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 00:07:35.841070 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:35.840983 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" podUID="7a3fc7b1-2b51-4009-ade1-d8b2c609a20c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 00:07:35.912310 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:35.912288 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:36.028938 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:36.028897 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:36.080825 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:36.080803 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:36.804416 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:36.804381 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:37.133345 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:37.133265 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:07:39.041885 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:39.041847 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:40.028090 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:40.028055 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:07:40.226108 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:40.226080 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:40.724980 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:40.724952 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:40.825039 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:40.824984 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:41.038649 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:41.038620 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:41.198875 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:41.198836 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:43.344774 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:43.344736 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:43.445711 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:43.445675 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:43.699875 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:43.699793 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:44.674719 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:44.674641 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:44.776110 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:44.776065 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:45.541299 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:45.541265 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:45.642309 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:45.642257 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:45.743181 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:45.743148 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:45.841845 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:45.841774 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:07:49.101087 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:49.101052 2570 generic.go:356] "Generic (PLEG): container finished" podID="f076d5df-065e-4fa3-b3c2-341bd58b1c1d" containerID="f2ffa648e668974053596a148e601f22d327b18f9372b3b9f26e741082580003" exitCode=0 Aug 04 00:07:49.101551 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:49.101131 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-7b4h2" event={"ID":"f076d5df-065e-4fa3-b3c2-341bd58b1c1d","Type":"ContainerDied","Data":"f2ffa648e668974053596a148e601f22d327b18f9372b3b9f26e741082580003"} Aug 04 00:07:49.101620 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:49.101559 2570 scope.go:117] "RemoveContainer" containerID="f2ffa648e668974053596a148e601f22d327b18f9372b3b9f26e741082580003" Aug 04 00:07:49.950456 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:49.950427 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-88l8k_c3ee0892-91f7-4cab-8b96-829a660ab62d/dns/0.log" Aug 04 00:07:50.083361 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:50.083321 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:50.107180 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:50.107153 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-7b4h2" event={"ID":"f076d5df-065e-4fa3-b3c2-341bd58b1c1d","Type":"ContainerStarted","Data":"e348ed3e0f6ba9827107dcb24c7b038467bab8ebf116d6bdded12b92870c6608"} Aug 04 00:07:50.150091 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:50.150070 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-88l8k_c3ee0892-91f7-4cab-8b96-829a660ab62d/kube-rbac-proxy/0.log" Aug 04 00:07:50.195201 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:50.195180 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:50.950031 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:50.950008 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mbkx7_fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117/dns-node-resolver/0.log" Aug 04 00:07:51.551160 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:51.551134 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fdgzw_df13c2f8-df07-47cb-82dc-2ac530b6985f/serve-healthcheck-canary/0.log" Aug 04 00:07:55.665735 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:55.665696 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:55.766950 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:55.766917 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:59.087920 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:59.087885 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:07:59.148097 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:07:59.148070 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:08:00.855688 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:00.855625 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" podUID="7a3fc7b1-2b51-4009-ade1-d8b2c609a20c" containerName="registry" containerID="cri-o://ce54e2f3ed792acb694ee57afd9b40c83601ea10258f7bb543e1fb39a1a8d751" gracePeriod=30 Aug 04 00:08:01.126996 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.126967 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:08:01.158818 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.158792 2570 generic.go:356] "Generic (PLEG): container finished" podID="7a3fc7b1-2b51-4009-ade1-d8b2c609a20c" containerID="ce54e2f3ed792acb694ee57afd9b40c83601ea10258f7bb543e1fb39a1a8d751" exitCode=0 Aug 04 00:08:01.158949 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.158855 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" event={"ID":"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c","Type":"ContainerDied","Data":"ce54e2f3ed792acb694ee57afd9b40c83601ea10258f7bb543e1fb39a1a8d751"} Aug 04 00:08:01.158949 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.158862 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" Aug 04 00:08:01.158949 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.158894 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc48d6bdf-m8vf4" event={"ID":"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c","Type":"ContainerDied","Data":"eb6f6f161482695c5bf9d3a5a8d48a48d004556fa3c81056bbed842e21f2caea"} Aug 04 00:08:01.158949 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.158915 2570 scope.go:117] "RemoveContainer" containerID="ce54e2f3ed792acb694ee57afd9b40c83601ea10258f7bb543e1fb39a1a8d751" Aug 04 00:08:01.167790 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.167772 2570 scope.go:117] "RemoveContainer" containerID="ce54e2f3ed792acb694ee57afd9b40c83601ea10258f7bb543e1fb39a1a8d751" Aug 04 00:08:01.168035 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:08:01.168014 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce54e2f3ed792acb694ee57afd9b40c83601ea10258f7bb543e1fb39a1a8d751\": container with ID starting with ce54e2f3ed792acb694ee57afd9b40c83601ea10258f7bb543e1fb39a1a8d751 not found: ID does not exist" containerID="ce54e2f3ed792acb694ee57afd9b40c83601ea10258f7bb543e1fb39a1a8d751" Aug 04 00:08:01.168115 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.168047 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce54e2f3ed792acb694ee57afd9b40c83601ea10258f7bb543e1fb39a1a8d751"} err="failed to get container status \"ce54e2f3ed792acb694ee57afd9b40c83601ea10258f7bb543e1fb39a1a8d751\": rpc error: code = NotFound desc = could not find container \"ce54e2f3ed792acb694ee57afd9b40c83601ea10258f7bb543e1fb39a1a8d751\": container with ID starting with ce54e2f3ed792acb694ee57afd9b40c83601ea10258f7bb543e1fb39a1a8d751 not found: ID does not exist" Aug 04 00:08:01.168304 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.168288 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-trusted-ca\") pod \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " Aug 04 00:08:01.168383 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.168333 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-certificates\") pod \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " Aug 04 00:08:01.168441 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.168393 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-image-registry-private-configuration\") pod \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " Aug 04 00:08:01.168492 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.168441 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls\") pod \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " Aug 04 00:08:01.168492 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.168465 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw7nx\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-kube-api-access-cw7nx\") pod \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " Aug 04 00:08:01.168659 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.168644 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-ca-trust-extracted\") pod \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " Aug 04 00:08:01.168911 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.168734 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c" (UID: "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:08:01.168911 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.168763 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c" (UID: "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:08:01.168911 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.168740 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-bound-sa-token\") pod \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " Aug 04 00:08:01.168911 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.168841 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-installation-pull-secrets\") pod \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\" (UID: \"7a3fc7b1-2b51-4009-ade1-d8b2c609a20c\") " Aug 04 00:08:01.169381 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.169356 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-trusted-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:08:01.169477 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.169387 2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-certificates\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:08:01.171728 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.171607 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c" (UID: "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:08:01.171728 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.171674 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c" (UID: "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:08:01.171890 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.171794 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-kube-api-access-cw7nx" (OuterVolumeSpecName: "kube-api-access-cw7nx") pod "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c" (UID: "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c"). InnerVolumeSpecName "kube-api-access-cw7nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:08:01.171890 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.171853 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c" (UID: "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:08:01.172675 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.172646 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c" (UID: "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:08:01.180838 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.180809 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c" (UID: "7a3fc7b1-2b51-4009-ade1-d8b2c609a20c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:08:01.270631 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.270601 2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-installation-pull-secrets\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:08:01.270631 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.270626 2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-image-registry-private-configuration\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:08:01.270631 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.270637 2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-registry-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:08:01.270844 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.270646 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw7nx\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-kube-api-access-cw7nx\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:08:01.270844 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.270655 2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-ca-trust-extracted\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:08:01.270844 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.270663 2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c-bound-sa-token\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:08:01.480816 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.480789 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fc48d6bdf-m8vf4"] Aug 04 00:08:01.484608 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:01.484586 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7fc48d6bdf-m8vf4"] Aug 04 00:08:02.033879 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:02.033841 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a3fc7b1-2b51-4009-ade1-d8b2c609a20c" path="/var/lib/kubelet/pods/7a3fc7b1-2b51-4009-ade1-d8b2c609a20c/volumes" Aug 04 00:08:04.173178 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:04.173111 2570 generic.go:356] "Generic (PLEG): container finished" podID="68bf8531-7950-4913-8b4b-ada9c386a6a3" containerID="ed4b6e28f010064a8a72013f9cde36d57ad70dfa4cd9fdfcf427dd0436f4134d" exitCode=0 Aug 04 00:08:04.173494 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:04.173184 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z" event={"ID":"68bf8531-7950-4913-8b4b-ada9c386a6a3","Type":"ContainerDied","Data":"ed4b6e28f010064a8a72013f9cde36d57ad70dfa4cd9fdfcf427dd0436f4134d"} Aug 04 00:08:04.173494 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:04.173484 2570 scope.go:117] "RemoveContainer" containerID="ed4b6e28f010064a8a72013f9cde36d57ad70dfa4cd9fdfcf427dd0436f4134d" Aug 04 00:08:05.179522 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:05.179486 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-z2j5z" event={"ID":"68bf8531-7950-4913-8b4b-ada9c386a6a3","Type":"ContainerStarted","Data":"cba7b6880a6f50d4a88e8322ffa42ec435ad7513e8ba8091ee3053239cd18633"} Aug 04 00:08:11.206210 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:11.206174 2570 generic.go:356] "Generic (PLEG): container finished" podID="2f998954-55dc-4e58-a70a-ad878047d419" containerID="8e3f015fdf89697b6062f6ce8a3080fd52b650f4bec79ae35f58de36ded541a8" exitCode=0 Aug 04 00:08:11.206681 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:11.206255 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29763360-vng42" event={"ID":"2f998954-55dc-4e58-a70a-ad878047d419","Type":"ContainerDied","Data":"8e3f015fdf89697b6062f6ce8a3080fd52b650f4bec79ae35f58de36ded541a8"} Aug 04 00:08:12.340817 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:12.340793 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29763360-vng42" Aug 04 00:08:12.460436 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:12.460383 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjj54\" (UniqueName: \"kubernetes.io/projected/2f998954-55dc-4e58-a70a-ad878047d419-kube-api-access-kjj54\") pod \"2f998954-55dc-4e58-a70a-ad878047d419\" (UID: \"2f998954-55dc-4e58-a70a-ad878047d419\") " Aug 04 00:08:12.460545 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:12.460436 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f998954-55dc-4e58-a70a-ad878047d419-serviceca\") pod \"2f998954-55dc-4e58-a70a-ad878047d419\" (UID: \"2f998954-55dc-4e58-a70a-ad878047d419\") " Aug 04 00:08:12.460785 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:12.460752 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f998954-55dc-4e58-a70a-ad878047d419-serviceca" (OuterVolumeSpecName: "serviceca") pod "2f998954-55dc-4e58-a70a-ad878047d419" (UID: "2f998954-55dc-4e58-a70a-ad878047d419"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:08:12.462410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:12.462390 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f998954-55dc-4e58-a70a-ad878047d419-kube-api-access-kjj54" (OuterVolumeSpecName: "kube-api-access-kjj54") pod "2f998954-55dc-4e58-a70a-ad878047d419" (UID: "2f998954-55dc-4e58-a70a-ad878047d419"). InnerVolumeSpecName "kube-api-access-kjj54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:08:12.561852 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:12.561826 2570 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f998954-55dc-4e58-a70a-ad878047d419-serviceca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:08:12.561852 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:12.561848 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjj54\" (UniqueName: \"kubernetes.io/projected/2f998954-55dc-4e58-a70a-ad878047d419-kube-api-access-kjj54\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:08:13.214902 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:13.214875 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29763360-vng42" Aug 04 00:08:13.215063 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:13.214916 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29763360-vng42" event={"ID":"2f998954-55dc-4e58-a70a-ad878047d419","Type":"ContainerDied","Data":"0cff42b3d0381869b1b132db19288070646f92fb9764cf779e0c1c8c77adde2b"} Aug 04 00:08:13.215063 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:13.214956 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cff42b3d0381869b1b132db19288070646f92fb9764cf779e0c1c8c77adde2b" Aug 04 00:08:18.819179 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:18.819142 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:08:18.916527 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:18.916500 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:08:18.968804 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:18.968778 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:08:27.133974 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:27.133879 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:08:27.153860 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:27.153824 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:08:27.288112 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:27.288089 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 00:08:36.730353 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:36.730308 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:08:36.805430 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:36.805395 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:08:47.760836 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:47.760805 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs\") pod \"network-metrics-daemon-9b87x\" (UID: \"e656f284-112c-4309-9151-41e5a9f45ede\") " pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:08:47.763051 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:47.763031 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e656f284-112c-4309-9151-41e5a9f45ede-metrics-certs\") pod \"network-metrics-daemon-9b87x\" (UID: \"e656f284-112c-4309-9151-41e5a9f45ede\") " pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:08:47.831418 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:47.831390 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f9tct\"" Aug 04 00:08:47.839487 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:47.839469 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b87x" Aug 04 00:08:47.960606 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:47.960578 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9b87x"] Aug 04 00:08:47.962203 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:08:47.962181 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode656f284_112c_4309_9151_41e5a9f45ede.slice/crio-bb4f946d5a8d4a8eb8b146948075a0604dc76637a747033c3feaf613dc477eef WatchSource:0}: Error finding container bb4f946d5a8d4a8eb8b146948075a0604dc76637a747033c3feaf613dc477eef: Status 404 returned error can't find the container with id bb4f946d5a8d4a8eb8b146948075a0604dc76637a747033c3feaf613dc477eef Aug 04 00:08:47.964006 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:47.963987 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 00:08:48.354705 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:48.354668 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9b87x" event={"ID":"e656f284-112c-4309-9151-41e5a9f45ede","Type":"ContainerStarted","Data":"bb4f946d5a8d4a8eb8b146948075a0604dc76637a747033c3feaf613dc477eef"} Aug 04 00:08:48.664629 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:48.664560 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:08:48.764507 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:48.764468 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:08:49.360833 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:49.360757 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9b87x" event={"ID":"e656f284-112c-4309-9151-41e5a9f45ede","Type":"ContainerStarted","Data":"6b9ce73474f41fba216e7475a064544f9085117055c1c716a32659bbbc9123b5"} Aug 04 00:08:49.360833 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:49.360810 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9b87x" event={"ID":"e656f284-112c-4309-9151-41e5a9f45ede","Type":"ContainerStarted","Data":"1ff6c18b492902abf0e598cb4664713fa0a19a1af730d71a28674c84ef11f07a"} Aug 04 00:08:49.376799 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:49.376754 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9b87x" podStartSLOduration=374.235967361 podStartE2EDuration="6m15.37674085s" totalImagesPullingTime="1.140773489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:08:49.375849366 +0000 UTC m=+375.858515696" watchObservedRunningTime="2026-08-04 00:08:49.37674085 +0000 UTC m=+375.859407181" Aug 04 00:08:49.467261 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:49.467218 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:08:49.579403 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:49.579377 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:08:49.639336 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:49.639284 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:08:55.673561 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:55.673517 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:08:55.773913 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:55.773883 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:08:56.982204 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:56.982169 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:08:57.078342 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:57.078315 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:08:57.177988 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:57.177961 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:08:57.902214 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:57.902181 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:08:57.954438 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:08:57.954412 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:09:07.864377 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:09:07.864339 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:09:08.092953 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:09:08.092912 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:09:08.152906 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:09:08.152836 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:09:19.647519 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:09:19.647478 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:09:19.702960 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:09:19.702928 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:09:37.717159 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:09:37.717125 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:03.129336 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:03.129218 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:03.228509 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:03.228471 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:29.978687 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:29.978642 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:30.810981 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:30.810949 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:33.575306 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:33.575275 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:33.674049 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:33.674011 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:35.222127 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:35.222093 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:35.323360 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:35.323324 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:35.435440 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:35.435403 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:35.539352 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:35.539322 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:37.605839 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:37.605807 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:38.040856 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:38.040821 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:40.377475 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:40.377441 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:40.438428 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:40.438392 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:44.812204 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:44.812172 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:47.332393 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.332352 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6r98l"] Aug 04 00:10:47.332767 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.332721 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f998954-55dc-4e58-a70a-ad878047d419" containerName="image-pruner" Aug 04 00:10:47.332767 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.332734 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f998954-55dc-4e58-a70a-ad878047d419" containerName="image-pruner" Aug 04 00:10:47.332767 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.332745 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a3fc7b1-2b51-4009-ade1-d8b2c609a20c" containerName="registry" Aug 04 00:10:47.332767 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.332751 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a3fc7b1-2b51-4009-ade1-d8b2c609a20c" containerName="registry" Aug 04 00:10:47.332889 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.332836 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a3fc7b1-2b51-4009-ade1-d8b2c609a20c" containerName="registry" Aug 04 00:10:47.332889 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.332850 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f998954-55dc-4e58-a70a-ad878047d419" containerName="image-pruner" Aug 04 00:10:47.335818 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.335800 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6r98l" Aug 04 00:10:47.338211 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.338190 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 04 00:10:47.338979 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.338958 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6t2lz\"" Aug 04 00:10:47.339096 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.338962 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 04 00:10:47.345671 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.345646 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6r98l"] Aug 04 00:10:47.359503 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.359474 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp8d6\" (UniqueName: \"kubernetes.io/projected/0653a706-0eeb-44fe-967a-c804bb025fa0-kube-api-access-tp8d6\") pod \"cert-manager-55b4d954d5-6r98l\" (UID: \"0653a706-0eeb-44fe-967a-c804bb025fa0\") " pod="cert-manager/cert-manager-55b4d954d5-6r98l" Aug 04 00:10:47.359652 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.359530 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0653a706-0eeb-44fe-967a-c804bb025fa0-bound-sa-token\") pod \"cert-manager-55b4d954d5-6r98l\" (UID: \"0653a706-0eeb-44fe-967a-c804bb025fa0\") " pod="cert-manager/cert-manager-55b4d954d5-6r98l" Aug 04 00:10:47.460361 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.460301 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp8d6\" (UniqueName: \"kubernetes.io/projected/0653a706-0eeb-44fe-967a-c804bb025fa0-kube-api-access-tp8d6\") pod \"cert-manager-55b4d954d5-6r98l\" (UID: \"0653a706-0eeb-44fe-967a-c804bb025fa0\") " pod="cert-manager/cert-manager-55b4d954d5-6r98l" Aug 04 00:10:47.460557 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.460384 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0653a706-0eeb-44fe-967a-c804bb025fa0-bound-sa-token\") pod \"cert-manager-55b4d954d5-6r98l\" (UID: \"0653a706-0eeb-44fe-967a-c804bb025fa0\") " pod="cert-manager/cert-manager-55b4d954d5-6r98l" Aug 04 00:10:47.469749 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.469712 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0653a706-0eeb-44fe-967a-c804bb025fa0-bound-sa-token\") pod \"cert-manager-55b4d954d5-6r98l\" (UID: \"0653a706-0eeb-44fe-967a-c804bb025fa0\") " pod="cert-manager/cert-manager-55b4d954d5-6r98l" Aug 04 00:10:47.469749 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.469736 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp8d6\" (UniqueName: \"kubernetes.io/projected/0653a706-0eeb-44fe-967a-c804bb025fa0-kube-api-access-tp8d6\") pod \"cert-manager-55b4d954d5-6r98l\" (UID: \"0653a706-0eeb-44fe-967a-c804bb025fa0\") " pod="cert-manager/cert-manager-55b4d954d5-6r98l" Aug 04 00:10:47.655958 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.655857 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6r98l" Aug 04 00:10:47.787459 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.787285 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6r98l"] Aug 04 00:10:47.790651 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:10:47.790617 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0653a706_0eeb_44fe_967a_c804bb025fa0.slice/crio-150af5e43f9f15b80be4b33dd23dc79c165ee0e9f08f6ed35b18d38c11bdb10e WatchSource:0}: Error finding container 150af5e43f9f15b80be4b33dd23dc79c165ee0e9f08f6ed35b18d38c11bdb10e: Status 404 returned error can't find the container with id 150af5e43f9f15b80be4b33dd23dc79c165ee0e9f08f6ed35b18d38c11bdb10e Aug 04 00:10:47.850635 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:47.850601 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6r98l" event={"ID":"0653a706-0eeb-44fe-967a-c804bb025fa0","Type":"ContainerStarted","Data":"150af5e43f9f15b80be4b33dd23dc79c165ee0e9f08f6ed35b18d38c11bdb10e"} Aug 04 00:10:51.371125 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:51.371090 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:51.471879 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:51.471848 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:54.820184 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:54.820146 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:10:54.892706 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:10:54.892670 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:00.188298 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:00.188253 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:00.293747 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:00.293700 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:00.910425 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:00.910381 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6r98l" event={"ID":"0653a706-0eeb-44fe-967a-c804bb025fa0","Type":"ContainerStarted","Data":"c9f614f085ec5652d7bc256a39dae282e17090ce7995c3a8cbdfb0a4f608c76c"} Aug 04 00:11:00.928212 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:00.928160 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-6r98l" podStartSLOduration=1.616283339 podStartE2EDuration="13.928142083s" totalImagesPullingTime="12.311858744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:11:00.927538012 +0000 UTC m=+507.410204341" watchObservedRunningTime="2026-08-04 00:11:00.928142083 +0000 UTC m=+507.410808414" Aug 04 00:11:01.877192 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:01.877134 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:06.098529 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.098490 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:06.141055 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.141022 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb"] Aug 04 00:11:06.144575 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.144550 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" Aug 04 00:11:06.147042 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.147017 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 04 00:11:06.147042 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.147036 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 04 00:11:06.147951 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.147922 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 04 00:11:06.148095 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.147937 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tp5bz\"" Aug 04 00:11:06.148095 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.147989 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 04 00:11:06.148095 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.148023 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 04 00:11:06.152489 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.152459 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb"] Aug 04 00:11:06.180501 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.180468 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:06.235946 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.235907 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5c4f\" (UniqueName: \"kubernetes.io/projected/f8bdb018-ec6e-4409-a279-0703521c9882-kube-api-access-h5c4f\") pod \"lws-controller-manager-7cfb669bf4-sd7rb\" (UID: \"f8bdb018-ec6e-4409-a279-0703521c9882\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" Aug 04 00:11:06.235946 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.235952 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f8bdb018-ec6e-4409-a279-0703521c9882-manager-config\") pod \"lws-controller-manager-7cfb669bf4-sd7rb\" (UID: \"f8bdb018-ec6e-4409-a279-0703521c9882\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" Aug 04 00:11:06.236156 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.236066 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8bdb018-ec6e-4409-a279-0703521c9882-metrics-cert\") pod \"lws-controller-manager-7cfb669bf4-sd7rb\" (UID: \"f8bdb018-ec6e-4409-a279-0703521c9882\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" Aug 04 00:11:06.236156 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.236120 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8bdb018-ec6e-4409-a279-0703521c9882-cert\") pod \"lws-controller-manager-7cfb669bf4-sd7rb\" (UID: \"f8bdb018-ec6e-4409-a279-0703521c9882\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" Aug 04 00:11:06.336662 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.336619 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8bdb018-ec6e-4409-a279-0703521c9882-metrics-cert\") pod \"lws-controller-manager-7cfb669bf4-sd7rb\" (UID: \"f8bdb018-ec6e-4409-a279-0703521c9882\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" Aug 04 00:11:06.336858 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.336678 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8bdb018-ec6e-4409-a279-0703521c9882-cert\") pod \"lws-controller-manager-7cfb669bf4-sd7rb\" (UID: \"f8bdb018-ec6e-4409-a279-0703521c9882\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" Aug 04 00:11:06.336858 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.336701 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5c4f\" (UniqueName: \"kubernetes.io/projected/f8bdb018-ec6e-4409-a279-0703521c9882-kube-api-access-h5c4f\") pod \"lws-controller-manager-7cfb669bf4-sd7rb\" (UID: \"f8bdb018-ec6e-4409-a279-0703521c9882\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" Aug 04 00:11:06.336858 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.336724 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f8bdb018-ec6e-4409-a279-0703521c9882-manager-config\") pod \"lws-controller-manager-7cfb669bf4-sd7rb\" (UID: \"f8bdb018-ec6e-4409-a279-0703521c9882\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" Aug 04 00:11:06.337444 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.337417 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f8bdb018-ec6e-4409-a279-0703521c9882-manager-config\") pod \"lws-controller-manager-7cfb669bf4-sd7rb\" (UID: \"f8bdb018-ec6e-4409-a279-0703521c9882\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" Aug 04 00:11:06.339194 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.339172 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8bdb018-ec6e-4409-a279-0703521c9882-metrics-cert\") pod \"lws-controller-manager-7cfb669bf4-sd7rb\" (UID: \"f8bdb018-ec6e-4409-a279-0703521c9882\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" Aug 04 00:11:06.339297 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.339201 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8bdb018-ec6e-4409-a279-0703521c9882-cert\") pod \"lws-controller-manager-7cfb669bf4-sd7rb\" (UID: \"f8bdb018-ec6e-4409-a279-0703521c9882\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" Aug 04 00:11:06.344624 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.344604 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5c4f\" (UniqueName: \"kubernetes.io/projected/f8bdb018-ec6e-4409-a279-0703521c9882-kube-api-access-h5c4f\") pod \"lws-controller-manager-7cfb669bf4-sd7rb\" (UID: \"f8bdb018-ec6e-4409-a279-0703521c9882\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" Aug 04 00:11:06.455784 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.455673 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" Aug 04 00:11:06.587107 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.587075 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb"] Aug 04 00:11:06.590659 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:11:06.590632 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bdb018_ec6e_4409_a279_0703521c9882.slice/crio-d8824ad709b3284c13b81f009458687edcf0df67322d79f2012bb45b95fb65cf WatchSource:0}: Error finding container d8824ad709b3284c13b81f009458687edcf0df67322d79f2012bb45b95fb65cf: Status 404 returned error can't find the container with id d8824ad709b3284c13b81f009458687edcf0df67322d79f2012bb45b95fb65cf Aug 04 00:11:06.938893 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:06.938859 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" event={"ID":"f8bdb018-ec6e-4409-a279-0703521c9882","Type":"ContainerStarted","Data":"d8824ad709b3284c13b81f009458687edcf0df67322d79f2012bb45b95fb65cf"} Aug 04 00:11:09.954211 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:09.954174 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" event={"ID":"f8bdb018-ec6e-4409-a279-0703521c9882","Type":"ContainerStarted","Data":"8aa04a121e826e9a6a30ce10358b71c0d6c3e6305c11c467a32a1f4eb418a3a7"} Aug 04 00:11:09.954620 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:09.954247 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" Aug 04 00:11:09.971723 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:09.971663 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" podStartSLOduration=1.457176633 podStartE2EDuration="3.971640486s" totalImagesPullingTime="2.514463853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:11:09.969962718 +0000 UTC m=+516.452629047" watchObservedRunningTime="2026-08-04 00:11:09.971640486 +0000 UTC m=+516.454306817" Aug 04 00:11:16.601339 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:16.601302 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:16.695137 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:16.695101 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:17.229644 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.229615 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:17.411947 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.411916 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:17.511711 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.511673 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:17.555299 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.555254 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq"] Aug 04 00:11:17.558994 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.558970 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:17.563953 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.563931 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 00:11:17.563953 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.563927 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 00:11:17.564155 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.563966 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h29qs\"" Aug 04 00:11:17.564155 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.564004 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 00:11:17.564155 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.563937 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 00:11:17.581614 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.581577 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq"] Aug 04 00:11:17.690044 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.690011 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:17.732800 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.732765 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/846f9d3e-5acd-47df-8904-76a6ff0d0781-tmp\") pod \"opendatahub-operator-controller-manager-67b98d597c-txvhq\" (UID: \"846f9d3e-5acd-47df-8904-76a6ff0d0781\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:17.732957 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.732811 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv6df\" (UniqueName: \"kubernetes.io/projected/846f9d3e-5acd-47df-8904-76a6ff0d0781-kube-api-access-pv6df\") pod \"opendatahub-operator-controller-manager-67b98d597c-txvhq\" (UID: \"846f9d3e-5acd-47df-8904-76a6ff0d0781\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:17.732957 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.732880 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/846f9d3e-5acd-47df-8904-76a6ff0d0781-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-txvhq\" (UID: \"846f9d3e-5acd-47df-8904-76a6ff0d0781\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:17.732957 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.732946 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/846f9d3e-5acd-47df-8904-76a6ff0d0781-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-txvhq\" (UID: \"846f9d3e-5acd-47df-8904-76a6ff0d0781\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:17.733114 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.732988 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/846f9d3e-5acd-47df-8904-76a6ff0d0781-manifests\") pod \"opendatahub-operator-controller-manager-67b98d597c-txvhq\" (UID: \"846f9d3e-5acd-47df-8904-76a6ff0d0781\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:17.800515 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.799297 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:17.834212 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.834174 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/846f9d3e-5acd-47df-8904-76a6ff0d0781-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-txvhq\" (UID: \"846f9d3e-5acd-47df-8904-76a6ff0d0781\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:17.834212 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.834217 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/846f9d3e-5acd-47df-8904-76a6ff0d0781-manifests\") pod \"opendatahub-operator-controller-manager-67b98d597c-txvhq\" (UID: \"846f9d3e-5acd-47df-8904-76a6ff0d0781\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:17.834482 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.834303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/846f9d3e-5acd-47df-8904-76a6ff0d0781-tmp\") pod \"opendatahub-operator-controller-manager-67b98d597c-txvhq\" (UID: \"846f9d3e-5acd-47df-8904-76a6ff0d0781\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:17.834482 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.834332 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv6df\" (UniqueName: \"kubernetes.io/projected/846f9d3e-5acd-47df-8904-76a6ff0d0781-kube-api-access-pv6df\") pod \"opendatahub-operator-controller-manager-67b98d597c-txvhq\" (UID: \"846f9d3e-5acd-47df-8904-76a6ff0d0781\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:17.834482 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.834370 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/846f9d3e-5acd-47df-8904-76a6ff0d0781-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-txvhq\" (UID: \"846f9d3e-5acd-47df-8904-76a6ff0d0781\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:17.834715 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.834691 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/846f9d3e-5acd-47df-8904-76a6ff0d0781-manifests\") pod \"opendatahub-operator-controller-manager-67b98d597c-txvhq\" (UID: \"846f9d3e-5acd-47df-8904-76a6ff0d0781\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:17.836770 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.836741 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/846f9d3e-5acd-47df-8904-76a6ff0d0781-tmp\") pod \"opendatahub-operator-controller-manager-67b98d597c-txvhq\" (UID: \"846f9d3e-5acd-47df-8904-76a6ff0d0781\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:17.836959 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.836928 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/846f9d3e-5acd-47df-8904-76a6ff0d0781-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-txvhq\" (UID: \"846f9d3e-5acd-47df-8904-76a6ff0d0781\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:17.837113 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.836938 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/846f9d3e-5acd-47df-8904-76a6ff0d0781-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-txvhq\" (UID: \"846f9d3e-5acd-47df-8904-76a6ff0d0781\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:17.844614 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.844582 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv6df\" (UniqueName: \"kubernetes.io/projected/846f9d3e-5acd-47df-8904-76a6ff0d0781-kube-api-access-pv6df\") pod \"opendatahub-operator-controller-manager-67b98d597c-txvhq\" (UID: \"846f9d3e-5acd-47df-8904-76a6ff0d0781\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:17.869542 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.869506 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:17.901485 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.901453 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:17.996256 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:17.996209 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:18.021052 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:18.021022 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq"] Aug 04 00:11:18.024905 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:11:18.024869 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846f9d3e_5acd_47df_8904_76a6ff0d0781.slice/crio-c681ff398cb39cb22a879f512bcd6ea505451c8706dcae39845cd8549ffed21c WatchSource:0}: Error finding container c681ff398cb39cb22a879f512bcd6ea505451c8706dcae39845cd8549ffed21c: Status 404 returned error can't find the container with id c681ff398cb39cb22a879f512bcd6ea505451c8706dcae39845cd8549ffed21c Aug 04 00:11:18.057973 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:18.057892 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:18.157162 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:18.157128 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:18.997656 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:18.997606 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" event={"ID":"846f9d3e-5acd-47df-8904-76a6ff0d0781","Type":"ContainerStarted","Data":"c681ff398cb39cb22a879f512bcd6ea505451c8706dcae39845cd8549ffed21c"} Aug 04 00:11:20.961571 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:20.961540 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-sd7rb" Aug 04 00:11:21.009749 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:21.009704 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" event={"ID":"846f9d3e-5acd-47df-8904-76a6ff0d0781","Type":"ContainerStarted","Data":"9f00f26b12f9b604d17c5867d16fd3d4bead369f40ddcdd77587449187732c1b"} Aug 04 00:11:22.016103 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:22.016067 2570 generic.go:356] "Generic (PLEG): container finished" podID="846f9d3e-5acd-47df-8904-76a6ff0d0781" containerID="9f00f26b12f9b604d17c5867d16fd3d4bead369f40ddcdd77587449187732c1b" exitCode=0 Aug 04 00:11:22.016535 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:22.016154 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" event={"ID":"846f9d3e-5acd-47df-8904-76a6ff0d0781","Type":"ContainerDied","Data":"9f00f26b12f9b604d17c5867d16fd3d4bead369f40ddcdd77587449187732c1b"} Aug 04 00:11:23.023469 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:23.023433 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" event={"ID":"846f9d3e-5acd-47df-8904-76a6ff0d0781","Type":"ContainerStarted","Data":"d5336b4a0a645031e7e4c6bffc81ea09872ff7a2ca2cb95a5cd9451dbff29778"} Aug 04 00:11:23.023954 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:23.023588 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:23.048702 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:23.048632 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" podStartSLOduration=1.8086627389999999 podStartE2EDuration="6.048610859s" totalImagesPullingTime="3.216072666s" totalInitContainerRuntime="1.023875454s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:11:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:11:23.04785995 +0000 UTC m=+529.530526279" watchObservedRunningTime="2026-08-04 00:11:23.048610859 +0000 UTC m=+529.531277191" Aug 04 00:11:34.034203 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:34.034135 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-txvhq" Aug 04 00:11:36.215326 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:36.215289 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:36.333523 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:36.333485 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:36.902539 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:36.902505 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:36.958387 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:36.958346 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-554b4579cb-m2ttw"] Aug 04 00:11:36.969334 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:36.969303 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-554b4579cb-m2ttw" Aug 04 00:11:36.973040 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:36.973011 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-554b4579cb-m2ttw"] Aug 04 00:11:36.973040 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:36.973032 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 04 00:11:36.973287 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:36.973194 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fdf5h\"" Aug 04 00:11:36.973411 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:36.973285 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 04 00:11:36.973411 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:36.973287 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 00:11:36.973546 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:36.973452 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 00:11:37.010565 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:37.010523 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75898a0-3ed1-47e7-b54e-6ce840578c96-tls-certs\") pod \"kube-auth-proxy-554b4579cb-m2ttw\" (UID: \"f75898a0-3ed1-47e7-b54e-6ce840578c96\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-m2ttw" Aug 04 00:11:37.010740 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:37.010581 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f75898a0-3ed1-47e7-b54e-6ce840578c96-tmp\") pod \"kube-auth-proxy-554b4579cb-m2ttw\" (UID: \"f75898a0-3ed1-47e7-b54e-6ce840578c96\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-m2ttw" Aug 04 00:11:37.010740 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:37.010605 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx8kn\" (UniqueName: \"kubernetes.io/projected/f75898a0-3ed1-47e7-b54e-6ce840578c96-kube-api-access-sx8kn\") pod \"kube-auth-proxy-554b4579cb-m2ttw\" (UID: \"f75898a0-3ed1-47e7-b54e-6ce840578c96\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-m2ttw" Aug 04 00:11:37.056000 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:37.055955 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:37.111984 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:37.111930 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f75898a0-3ed1-47e7-b54e-6ce840578c96-tmp\") pod \"kube-auth-proxy-554b4579cb-m2ttw\" (UID: \"f75898a0-3ed1-47e7-b54e-6ce840578c96\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-m2ttw" Aug 04 00:11:37.112178 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:37.111991 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx8kn\" (UniqueName: \"kubernetes.io/projected/f75898a0-3ed1-47e7-b54e-6ce840578c96-kube-api-access-sx8kn\") pod \"kube-auth-proxy-554b4579cb-m2ttw\" (UID: \"f75898a0-3ed1-47e7-b54e-6ce840578c96\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-m2ttw" Aug 04 00:11:37.112178 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:37.112095 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75898a0-3ed1-47e7-b54e-6ce840578c96-tls-certs\") pod \"kube-auth-proxy-554b4579cb-m2ttw\" (UID: \"f75898a0-3ed1-47e7-b54e-6ce840578c96\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-m2ttw" Aug 04 00:11:37.114797 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:37.114770 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f75898a0-3ed1-47e7-b54e-6ce840578c96-tmp\") pod \"kube-auth-proxy-554b4579cb-m2ttw\" (UID: \"f75898a0-3ed1-47e7-b54e-6ce840578c96\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-m2ttw" Aug 04 00:11:37.114960 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:37.114938 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75898a0-3ed1-47e7-b54e-6ce840578c96-tls-certs\") pod \"kube-auth-proxy-554b4579cb-m2ttw\" (UID: \"f75898a0-3ed1-47e7-b54e-6ce840578c96\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-m2ttw" Aug 04 00:11:37.120758 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:37.120733 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx8kn\" (UniqueName: \"kubernetes.io/projected/f75898a0-3ed1-47e7-b54e-6ce840578c96-kube-api-access-sx8kn\") pod \"kube-auth-proxy-554b4579cb-m2ttw\" (UID: \"f75898a0-3ed1-47e7-b54e-6ce840578c96\") " pod="openshift-ingress/kube-auth-proxy-554b4579cb-m2ttw" Aug 04 00:11:37.122086 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:37.122062 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:37.223450 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:37.223368 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:37.282404 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:37.282369 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-554b4579cb-m2ttw" Aug 04 00:11:37.330281 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:37.330251 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:37.432970 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:37.432926 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:37.433193 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:37.432991 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-554b4579cb-m2ttw"] Aug 04 00:11:37.436027 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:11:37.435990 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf75898a0_3ed1_47e7_b54e_6ce840578c96.slice/crio-a21e9e8226cef2de4ca301a2e8136f69cd9cd3c810a7d98ed2445d9074ba9001 WatchSource:0}: Error finding container a21e9e8226cef2de4ca301a2e8136f69cd9cd3c810a7d98ed2445d9074ba9001: Status 404 returned error can't find the container with id a21e9e8226cef2de4ca301a2e8136f69cd9cd3c810a7d98ed2445d9074ba9001 Aug 04 00:11:37.953377 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:37.953341 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:38.096688 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:38.096502 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-554b4579cb-m2ttw" event={"ID":"f75898a0-3ed1-47e7-b54e-6ce840578c96","Type":"ContainerStarted","Data":"a21e9e8226cef2de4ca301a2e8136f69cd9cd3c810a7d98ed2445d9074ba9001"} Aug 04 00:11:38.209876 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:38.209770 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:38.377395 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:38.377325 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:38.440363 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:38.440290 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:38.876958 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:38.876864 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:38.944998 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:38.944964 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:39.400698 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:39.400652 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:41.116686 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:41.116646 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-554b4579cb-m2ttw" event={"ID":"f75898a0-3ed1-47e7-b54e-6ce840578c96","Type":"ContainerStarted","Data":"360c6aa15515877b0c9bc52c75ddc6144886edf78907861ae9055a2ab42ecda1"} Aug 04 00:11:41.133016 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:41.132960 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-554b4579cb-m2ttw" podStartSLOduration=2.249577284 podStartE2EDuration="5.132943253s" totalImagesPullingTime="2.883365969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:11:41.132018916 +0000 UTC m=+547.614685245" watchObservedRunningTime="2026-08-04 00:11:41.132943253 +0000 UTC m=+547.615609583" Aug 04 00:11:46.676411 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:46.676359 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:46.769619 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:46.769583 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:47.270031 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:47.269994 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:47.361534 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:47.361498 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:47.468539 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:47.468405 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:47.585803 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:47.585719 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:47.719257 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:47.719207 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:47.875560 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:47.875471 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:48.075809 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:48.075775 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:48.623215 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:48.623187 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:48.742797 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:48.742761 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:11:48.830118 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:11:48.830084 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:13.132115 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:13.132077 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:14.556136 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:14.556097 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:24.730680 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:24.730647 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:24.834946 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:24.834910 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:24.936074 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:24.936032 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:33.953815 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:33.953787 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:12:33.954308 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:33.954291 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:12:33.956700 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:33.956676 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:12:33.956916 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:33.956900 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:12:33.961720 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:33.961691 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:12:33.962135 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:33.962117 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:12:37.922278 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:37.922216 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:38.022031 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:38.021992 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:49.133714 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:49.133676 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:49.988156 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:49.988103 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:53.157504 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:53.157471 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:53.387345 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:53.387314 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:53.487784 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:53.487757 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:54.882836 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:54.882803 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:55.539464 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:55.539414 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:56.183433 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:56.183403 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:57.540063 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:57.540033 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:57.936857 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:57.936769 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:58.347672 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:58.347635 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:58.820639 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:58.820593 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:58.915691 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:58.915636 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:12:59.919658 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:12:59.919608 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:02.955406 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:02.955355 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:03.056473 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:03.056439 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:03.552737 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:03.552703 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:03.659873 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:03.659833 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:05.946238 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:05.946175 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:05.998427 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:05.998392 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:08.709427 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:08.709383 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:08.709975 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:08.709953 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd"] Aug 04 00:13:08.713320 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:08.713301 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd" Aug 04 00:13:08.715790 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:08.715766 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 00:13:08.715932 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:08.715767 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Aug 04 00:13:08.716595 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:08.716569 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Aug 04 00:13:08.716721 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:08.716652 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rlnxv\"" Aug 04 00:13:08.716721 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:08.716664 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 00:13:08.721312 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:08.721109 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd"] Aug 04 00:13:08.760565 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:08.760526 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ae3da25-da2a-425e-b498-302e6198d7f3-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-b9hhd\" (UID: \"7ae3da25-da2a-425e-b498-302e6198d7f3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd" Aug 04 00:13:08.760770 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:08.760613 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgs7x\" (UniqueName: \"kubernetes.io/projected/7ae3da25-da2a-425e-b498-302e6198d7f3-kube-api-access-hgs7x\") pod \"kuadrant-console-plugin-598f86468b-b9hhd\" (UID: \"7ae3da25-da2a-425e-b498-302e6198d7f3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd" Aug 04 00:13:08.760770 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:08.760649 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ae3da25-da2a-425e-b498-302e6198d7f3-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-b9hhd\" (UID: \"7ae3da25-da2a-425e-b498-302e6198d7f3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd" Aug 04 00:13:08.862014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:08.861972 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgs7x\" (UniqueName: \"kubernetes.io/projected/7ae3da25-da2a-425e-b498-302e6198d7f3-kube-api-access-hgs7x\") pod \"kuadrant-console-plugin-598f86468b-b9hhd\" (UID: \"7ae3da25-da2a-425e-b498-302e6198d7f3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd" Aug 04 00:13:08.862014 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:08.862025 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ae3da25-da2a-425e-b498-302e6198d7f3-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-b9hhd\" (UID: \"7ae3da25-da2a-425e-b498-302e6198d7f3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd" Aug 04 00:13:08.862312 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:08.862070 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ae3da25-da2a-425e-b498-302e6198d7f3-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-b9hhd\" (UID: \"7ae3da25-da2a-425e-b498-302e6198d7f3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd" Aug 04 00:13:08.862312 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:13:08.862204 2570 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Aug 04 00:13:08.862417 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:13:08.862315 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ae3da25-da2a-425e-b498-302e6198d7f3-plugin-serving-cert podName:7ae3da25-da2a-425e-b498-302e6198d7f3 nodeName:}" failed. No retries permitted until 2026-08-04 00:13:09.362296328 +0000 UTC m=+635.844962636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/7ae3da25-da2a-425e-b498-302e6198d7f3-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-b9hhd" (UID: "7ae3da25-da2a-425e-b498-302e6198d7f3") : secret "plugin-serving-cert" not found Aug 04 00:13:08.862693 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:08.862675 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ae3da25-da2a-425e-b498-302e6198d7f3-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-b9hhd\" (UID: \"7ae3da25-da2a-425e-b498-302e6198d7f3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd" Aug 04 00:13:08.871631 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:08.871610 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgs7x\" (UniqueName: \"kubernetes.io/projected/7ae3da25-da2a-425e-b498-302e6198d7f3-kube-api-access-hgs7x\") pod \"kuadrant-console-plugin-598f86468b-b9hhd\" (UID: \"7ae3da25-da2a-425e-b498-302e6198d7f3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd" Aug 04 00:13:09.339495 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:09.339458 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:09.365819 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:09.365782 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ae3da25-da2a-425e-b498-302e6198d7f3-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-b9hhd\" (UID: \"7ae3da25-da2a-425e-b498-302e6198d7f3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd" Aug 04 00:13:09.368218 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:09.368195 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ae3da25-da2a-425e-b498-302e6198d7f3-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-b9hhd\" (UID: \"7ae3da25-da2a-425e-b498-302e6198d7f3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd" Aug 04 00:13:09.624217 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:09.624129 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd" Aug 04 00:13:09.763285 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:09.763249 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd"] Aug 04 00:13:09.766715 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:13:09.766680 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ae3da25_da2a_425e_b498_302e6198d7f3.slice/crio-34459833f212f18e4f111fc640469936f26b51645244eb0b131946d4da2597dd WatchSource:0}: Error finding container 34459833f212f18e4f111fc640469936f26b51645244eb0b131946d4da2597dd: Status 404 returned error can't find the container with id 34459833f212f18e4f111fc640469936f26b51645244eb0b131946d4da2597dd Aug 04 00:13:10.153550 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:10.153505 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:10.254862 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:10.254830 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:10.545024 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:10.544949 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd" event={"ID":"7ae3da25-da2a-425e-b498-302e6198d7f3","Type":"ContainerStarted","Data":"34459833f212f18e4f111fc640469936f26b51645244eb0b131946d4da2597dd"} Aug 04 00:13:14.897511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:14.897425 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:14.994852 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:14.994819 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:15.054574 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:15.054545 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:15.573533 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:15.573498 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd" event={"ID":"7ae3da25-da2a-425e-b498-302e6198d7f3","Type":"ContainerStarted","Data":"8d15e02f19e0d44a0cef617f7a1d32d5f46d99569a1defdd2cd66184e801dc7b"} Aug 04 00:13:15.591799 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:15.591753 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-b9hhd" podStartSLOduration=2.855976899 podStartE2EDuration="7.5917366s" totalImagesPullingTime="4.735759701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:13:15.591260527 +0000 UTC m=+642.073926857" watchObservedRunningTime="2026-08-04 00:13:15.5917366 +0000 UTC m=+642.074402963" Aug 04 00:13:16.560327 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:16.560289 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:16.621173 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:16.621136 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:23.004029 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:23.003992 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:23.057399 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:23.057359 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:23.599164 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:23.599121 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:23.662821 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:23.662791 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:26.608107 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:26.608070 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:26.665921 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:26.665884 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:31.203418 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:31.203376 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:31.307604 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:31.307575 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:34.961770 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:34.961701 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:35.061138 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:35.061105 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:35.161290 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:35.161214 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:37.799579 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:37.799537 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:47.662075 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:47.662043 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:47.875422 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:47.875388 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:56.876628 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:56.876597 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:56.929450 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:56.929414 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:59.704789 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:59.704733 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:59.810307 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:59.810275 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:13:59.910139 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:13:59.910108 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:14:00.009526 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.009486 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:14:00.012429 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.012406 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tws8j" Aug 04 00:14:00.014892 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.014871 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 04 00:14:00.027584 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.025934 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:14:00.045128 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.045097 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:14:00.116598 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.116557 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w64nx\" (UniqueName: \"kubernetes.io/projected/6a778ff6-d312-4fa6-9697-e71e03aef044-kube-api-access-w64nx\") pod \"limitador-limitador-85d9db4d5d-tws8j\" (UID: \"6a778ff6-d312-4fa6-9697-e71e03aef044\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tws8j" Aug 04 00:14:00.116810 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.116782 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6a778ff6-d312-4fa6-9697-e71e03aef044-config-file\") pod \"limitador-limitador-85d9db4d5d-tws8j\" (UID: \"6a778ff6-d312-4fa6-9697-e71e03aef044\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tws8j" Aug 04 00:14:00.218165 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.218125 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w64nx\" (UniqueName: \"kubernetes.io/projected/6a778ff6-d312-4fa6-9697-e71e03aef044-kube-api-access-w64nx\") pod \"limitador-limitador-85d9db4d5d-tws8j\" (UID: \"6a778ff6-d312-4fa6-9697-e71e03aef044\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tws8j" Aug 04 00:14:00.218359 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.218268 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6a778ff6-d312-4fa6-9697-e71e03aef044-config-file\") pod \"limitador-limitador-85d9db4d5d-tws8j\" (UID: \"6a778ff6-d312-4fa6-9697-e71e03aef044\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tws8j" Aug 04 00:14:00.218958 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.218936 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6a778ff6-d312-4fa6-9697-e71e03aef044-config-file\") pod \"limitador-limitador-85d9db4d5d-tws8j\" (UID: \"6a778ff6-d312-4fa6-9697-e71e03aef044\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tws8j" Aug 04 00:14:00.225816 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.225790 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w64nx\" (UniqueName: \"kubernetes.io/projected/6a778ff6-d312-4fa6-9697-e71e03aef044-kube-api-access-w64nx\") pod \"limitador-limitador-85d9db4d5d-tws8j\" (UID: \"6a778ff6-d312-4fa6-9697-e71e03aef044\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tws8j" Aug 04 00:14:00.325603 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.325515 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tws8j" Aug 04 00:14:00.458213 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.458052 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:14:00.461247 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:14:00.461191 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a778ff6_d312_4fa6_9697_e71e03aef044.slice/crio-f04bb5ec16451ac29706915c73776d0432fcc3ad8617d98acc41871ccb3835e1 WatchSource:0}: Error finding container f04bb5ec16451ac29706915c73776d0432fcc3ad8617d98acc41871ccb3835e1: Status 404 returned error can't find the container with id f04bb5ec16451ac29706915c73776d0432fcc3ad8617d98acc41871ccb3835e1 Aug 04 00:14:00.463143 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.463123 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 00:14:00.525584 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.525527 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:14:00.602631 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.602488 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:14:00.603235 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.603205 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6kqh6"] Aug 04 00:14:00.607815 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.607797 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-6kqh6" Aug 04 00:14:00.610050 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.610027 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zg8cz\"" Aug 04 00:14:00.614509 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.614485 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6kqh6"] Aug 04 00:14:00.722848 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.722810 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdgck\" (UniqueName: \"kubernetes.io/projected/98a29340-54b9-4ad4-950e-138405c3334f-kube-api-access-xdgck\") pod \"authorino-c548b5d7d-6kqh6\" (UID: \"98a29340-54b9-4ad4-950e-138405c3334f\") " pod="kuadrant-system/authorino-c548b5d7d-6kqh6" Aug 04 00:14:00.788151 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.788115 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tws8j" event={"ID":"6a778ff6-d312-4fa6-9697-e71e03aef044","Type":"ContainerStarted","Data":"f04bb5ec16451ac29706915c73776d0432fcc3ad8617d98acc41871ccb3835e1"} Aug 04 00:14:00.823623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.823581 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdgck\" (UniqueName: \"kubernetes.io/projected/98a29340-54b9-4ad4-950e-138405c3334f-kube-api-access-xdgck\") pod \"authorino-c548b5d7d-6kqh6\" (UID: \"98a29340-54b9-4ad4-950e-138405c3334f\") " pod="kuadrant-system/authorino-c548b5d7d-6kqh6" Aug 04 00:14:00.831760 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.831724 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdgck\" (UniqueName: \"kubernetes.io/projected/98a29340-54b9-4ad4-950e-138405c3334f-kube-api-access-xdgck\") pod \"authorino-c548b5d7d-6kqh6\" (UID: \"98a29340-54b9-4ad4-950e-138405c3334f\") " pod="kuadrant-system/authorino-c548b5d7d-6kqh6" Aug 04 00:14:00.918495 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:00.918409 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-6kqh6" Aug 04 00:14:01.048089 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:01.048061 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6kqh6"] Aug 04 00:14:01.050166 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:14:01.050140 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98a29340_54b9_4ad4_950e_138405c3334f.slice/crio-3734b4fad80be4125ca34b13ec7ea1c38c1b1a68ad341c21d25f0e5e92e720bf WatchSource:0}: Error finding container 3734b4fad80be4125ca34b13ec7ea1c38c1b1a68ad341c21d25f0e5e92e720bf: Status 404 returned error can't find the container with id 3734b4fad80be4125ca34b13ec7ea1c38c1b1a68ad341c21d25f0e5e92e720bf Aug 04 00:14:01.794747 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:01.794710 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-6kqh6" event={"ID":"98a29340-54b9-4ad4-950e-138405c3334f","Type":"ContainerStarted","Data":"3734b4fad80be4125ca34b13ec7ea1c38c1b1a68ad341c21d25f0e5e92e720bf"} Aug 04 00:14:03.807899 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:03.807864 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tws8j" event={"ID":"6a778ff6-d312-4fa6-9697-e71e03aef044","Type":"ContainerStarted","Data":"ab168d56c3143051b427449f53e5edba54255313f6be20f9d2b9c882a1b3564d"} Aug 04 00:14:03.808306 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:03.807927 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tws8j" Aug 04 00:14:03.822723 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:03.822673 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tws8j" podStartSLOduration=2.129302601 podStartE2EDuration="4.822656701s" totalImagesPullingTime="2.6933541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:13:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:14:03.822214859 +0000 UTC m=+690.304881191" watchObservedRunningTime="2026-08-04 00:14:03.822656701 +0000 UTC m=+690.305323032" Aug 04 00:14:06.823357 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:06.823321 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-6kqh6" event={"ID":"98a29340-54b9-4ad4-950e-138405c3334f","Type":"ContainerStarted","Data":"b6f2000a10c99e86cae940d07b6447023ab3f30cd7f57cb91a662b9a7a320950"} Aug 04 00:14:06.838753 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:06.838702 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-6kqh6" podStartSLOduration=1.441010108 podStartE2EDuration="6.838686121s" totalImagesPullingTime="5.397676013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:14:06.838287734 +0000 UTC m=+693.320954067" watchObservedRunningTime="2026-08-04 00:14:06.838686121 +0000 UTC m=+693.321352450" Aug 04 00:14:14.814528 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:14.814497 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tws8j" Aug 04 00:14:20.873758 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:20.873722 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:14:20.970994 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:20.970962 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:14:21.369067 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:21.369021 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:14:33.571490 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:33.571408 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:14:34.265179 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:34.265137 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:14:35.644409 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:35.644367 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6kqh6"] Aug 04 00:14:35.644846 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:35.644598 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-6kqh6" podUID="98a29340-54b9-4ad4-950e-138405c3334f" containerName="authorino" containerID="cri-o://b6f2000a10c99e86cae940d07b6447023ab3f30cd7f57cb91a662b9a7a320950" gracePeriod=30 Aug 04 00:14:35.888610 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:35.888579 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-6kqh6" Aug 04 00:14:35.953768 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:35.953680 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdgck\" (UniqueName: \"kubernetes.io/projected/98a29340-54b9-4ad4-950e-138405c3334f-kube-api-access-xdgck\") pod \"98a29340-54b9-4ad4-950e-138405c3334f\" (UID: \"98a29340-54b9-4ad4-950e-138405c3334f\") " Aug 04 00:14:35.955953 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:35.955926 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a29340-54b9-4ad4-950e-138405c3334f-kube-api-access-xdgck" (OuterVolumeSpecName: "kube-api-access-xdgck") pod "98a29340-54b9-4ad4-950e-138405c3334f" (UID: "98a29340-54b9-4ad4-950e-138405c3334f"). InnerVolumeSpecName "kube-api-access-xdgck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:14:35.959485 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:35.959458 2570 generic.go:356] "Generic (PLEG): container finished" podID="98a29340-54b9-4ad4-950e-138405c3334f" containerID="b6f2000a10c99e86cae940d07b6447023ab3f30cd7f57cb91a662b9a7a320950" exitCode=0 Aug 04 00:14:35.959627 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:35.959493 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-6kqh6" Aug 04 00:14:35.959627 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:35.959499 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-6kqh6" event={"ID":"98a29340-54b9-4ad4-950e-138405c3334f","Type":"ContainerDied","Data":"b6f2000a10c99e86cae940d07b6447023ab3f30cd7f57cb91a662b9a7a320950"} Aug 04 00:14:35.959627 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:35.959531 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-6kqh6" event={"ID":"98a29340-54b9-4ad4-950e-138405c3334f","Type":"ContainerDied","Data":"3734b4fad80be4125ca34b13ec7ea1c38c1b1a68ad341c21d25f0e5e92e720bf"} Aug 04 00:14:35.959627 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:35.959549 2570 scope.go:117] "RemoveContainer" containerID="b6f2000a10c99e86cae940d07b6447023ab3f30cd7f57cb91a662b9a7a320950" Aug 04 00:14:35.968556 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:35.968536 2570 scope.go:117] "RemoveContainer" containerID="b6f2000a10c99e86cae940d07b6447023ab3f30cd7f57cb91a662b9a7a320950" Aug 04 00:14:35.968882 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:14:35.968861 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6f2000a10c99e86cae940d07b6447023ab3f30cd7f57cb91a662b9a7a320950\": container with ID starting with b6f2000a10c99e86cae940d07b6447023ab3f30cd7f57cb91a662b9a7a320950 not found: ID does not exist" containerID="b6f2000a10c99e86cae940d07b6447023ab3f30cd7f57cb91a662b9a7a320950" Aug 04 00:14:35.968930 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:35.968892 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f2000a10c99e86cae940d07b6447023ab3f30cd7f57cb91a662b9a7a320950"} err="failed to get container status \"b6f2000a10c99e86cae940d07b6447023ab3f30cd7f57cb91a662b9a7a320950\": rpc error: code = NotFound desc = could not find container \"b6f2000a10c99e86cae940d07b6447023ab3f30cd7f57cb91a662b9a7a320950\": container with ID starting with b6f2000a10c99e86cae940d07b6447023ab3f30cd7f57cb91a662b9a7a320950 not found: ID does not exist" Aug 04 00:14:35.979800 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:35.979760 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6kqh6"] Aug 04 00:14:35.981659 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:35.981635 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6kqh6"] Aug 04 00:14:36.033926 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:36.033889 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98a29340-54b9-4ad4-950e-138405c3334f" path="/var/lib/kubelet/pods/98a29340-54b9-4ad4-950e-138405c3334f/volumes" Aug 04 00:14:36.054316 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:36.054287 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdgck\" (UniqueName: \"kubernetes.io/projected/98a29340-54b9-4ad4-950e-138405c3334f-kube-api-access-xdgck\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:14:40.262837 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:40.262794 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:14:40.356087 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:40.356026 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:14:40.431519 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:40.431474 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:14:40.594066 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:40.593910 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:14:43.430241 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:43.430198 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:14:43.531628 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:14:43.531586 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:15:02.262805 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:02.262768 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:15:15.140756 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:15.140703 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:15:15.237898 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:15.237861 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:15:16.743390 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:16.743350 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:15:17.156110 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:17.156019 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:15:17.257266 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:17.257217 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:15:17.335342 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:17.335313 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:15:17.380468 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:17.380432 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:15:27.487357 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:27.487317 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:15:27.622550 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:27.622510 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:15:32.865671 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:32.865636 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:15:33.326182 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:33.326145 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:15:46.819953 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:46.819916 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l"] Aug 04 00:15:46.820328 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:46.820318 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a29340-54b9-4ad4-950e-138405c3334f" containerName="authorino" Aug 04 00:15:46.820371 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:46.820332 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a29340-54b9-4ad4-950e-138405c3334f" containerName="authorino" Aug 04 00:15:46.820406 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:46.820396 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a29340-54b9-4ad4-950e-138405c3334f" containerName="authorino" Aug 04 00:15:46.825006 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:46.824981 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:46.827551 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:46.827523 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 04 00:15:46.827686 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:46.827552 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 04 00:15:46.828490 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:46.828467 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Aug 04 00:15:46.828596 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:46.828492 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k7knc\"" Aug 04 00:15:46.834153 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:46.834128 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l"] Aug 04 00:15:46.854491 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:46.854431 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:15:46.996301 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:46.996265 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:46.996511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:46.996323 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:46.996511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:46.996379 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl96f\" (UniqueName: \"kubernetes.io/projected/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-kube-api-access-hl96f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:46.996511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:46.996419 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:46.996511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:46.996479 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:46.996511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:46.996498 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:47.097312 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:47.097183 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:47.097312 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:47.097261 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:47.097312 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:47.097286 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl96f\" (UniqueName: \"kubernetes.io/projected/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-kube-api-access-hl96f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:47.097312 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:47.097304 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:47.097706 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:47.097384 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:47.097706 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:47.097421 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:47.097806 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:47.097713 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:47.097806 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:47.097733 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:47.097806 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:47.097778 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:47.099589 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:47.099563 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:47.099909 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:47.099893 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:47.105241 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:47.105198 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl96f\" (UniqueName: \"kubernetes.io/projected/7c7bd60b-7ecb-4c68-82fd-dee5993b9c51-kube-api-access-hl96f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzb7l\" (UID: \"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:47.137561 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:47.137525 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:47.278409 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:47.278376 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l"] Aug 04 00:15:47.282969 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:15:47.282932 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7bd60b_7ecb_4c68_82fd_dee5993b9c51.slice/crio-6d1043de9654754375ae5d83b7298460a6b965baca735711be14202ea6b9c14e WatchSource:0}: Error finding container 6d1043de9654754375ae5d83b7298460a6b965baca735711be14202ea6b9c14e: Status 404 returned error can't find the container with id 6d1043de9654754375ae5d83b7298460a6b965baca735711be14202ea6b9c14e Aug 04 00:15:47.305960 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:47.305928 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" event={"ID":"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51","Type":"ContainerStarted","Data":"6d1043de9654754375ae5d83b7298460a6b965baca735711be14202ea6b9c14e"} Aug 04 00:15:47.432643 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:47.432552 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:15:49.317871 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:49.317834 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" event={"ID":"7c7bd60b-7ecb-4c68-82fd-dee5993b9c51","Type":"ContainerStarted","Data":"fdb68d522871c55717f91449c7eaa018affba30dc06274ebb28eaf3fcd54d565"} Aug 04 00:15:49.335092 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:49.335040 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" podStartSLOduration=2.295940405 podStartE2EDuration="3.33502231s" totalImagesPullingTime="1.039081905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:15:49.333780955 +0000 UTC m=+795.816447285" watchObservedRunningTime="2026-08-04 00:15:49.33502231 +0000 UTC m=+795.817688673" Aug 04 00:15:51.222777 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.222739 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g"] Aug 04 00:15:51.226313 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.226290 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.228698 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.228679 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Aug 04 00:15:51.236888 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.236861 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g"] Aug 04 00:15:51.252427 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.252395 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:15:51.337500 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.337464 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52zgt\" (UniqueName: \"kubernetes.io/projected/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-kube-api-access-52zgt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.337500 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.337508 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.337722 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.337567 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.337722 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.337646 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.337722 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.337679 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.337722 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.337702 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.440994 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.440923 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52zgt\" (UniqueName: \"kubernetes.io/projected/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-kube-api-access-52zgt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.440994 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.441003 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.441617 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.441038 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.441617 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.441112 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.441617 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.441148 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.441617 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.441181 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.441617 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.441581 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.441820 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.441661 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.441820 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.441694 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.443481 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.443459 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.443721 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.443705 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.449808 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.449779 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52zgt\" (UniqueName: \"kubernetes.io/projected/f48c767b-2ca5-440f-97ec-7ec88a52c9aa-kube-api-access-52zgt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzk7g\" (UID: \"f48c767b-2ca5-440f-97ec-7ec88a52c9aa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.537468 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.537417 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:15:51.680697 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:51.680661 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g"] Aug 04 00:15:51.684909 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:15:51.684877 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48c767b_2ca5_440f_97ec_7ec88a52c9aa.slice/crio-a62726f8c4165ff4802779151b6f55c8f1d9e0f7f3fb53ea8d73e9e2375cfc78 WatchSource:0}: Error finding container a62726f8c4165ff4802779151b6f55c8f1d9e0f7f3fb53ea8d73e9e2375cfc78: Status 404 returned error can't find the container with id a62726f8c4165ff4802779151b6f55c8f1d9e0f7f3fb53ea8d73e9e2375cfc78 Aug 04 00:15:52.024290 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:52.024250 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:15:52.333687 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:52.333644 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" event={"ID":"f48c767b-2ca5-440f-97ec-7ec88a52c9aa","Type":"ContainerStarted","Data":"4cf1ec9590c8605cdfaf822f34bd010518f7f5eccacc6ae0a8e1569defa9d94b"} Aug 04 00:15:52.334152 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:52.333699 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" event={"ID":"f48c767b-2ca5-440f-97ec-7ec88a52c9aa","Type":"ContainerStarted","Data":"a62726f8c4165ff4802779151b6f55c8f1d9e0f7f3fb53ea8d73e9e2375cfc78"} Aug 04 00:15:52.374331 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:52.374268 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" podStartSLOduration=1.201700729 podStartE2EDuration="1.374250839s" totalImagesPullingTime="172.55011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:15:52.37292059 +0000 UTC m=+798.855586920" watchObservedRunningTime="2026-08-04 00:15:52.374250839 +0000 UTC m=+798.856917166" Aug 04 00:15:57.138434 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:57.138303 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:57.138434 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:57.138389 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:57.151854 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:57.151823 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:57.370036 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:57.370005 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzb7l" Aug 04 00:15:58.949160 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:15:58.949123 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:16:00.123847 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:16:00.123796 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:16:01.538382 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:16:01.538345 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:16:01.538801 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:16:01.538420 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:16:01.552701 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:16:01.552669 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:16:02.393668 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:16:02.393639 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzk7g" Aug 04 00:16:18.958345 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:16:18.958309 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:16:21.232992 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:16:21.232953 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:16:35.182135 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:16:35.182091 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:16:37.736462 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:16:37.736422 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:16:40.015704 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:16:40.015673 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:16:40.071543 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:16:40.071486 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:17:23.020955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:17:23.020917 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:17:28.753881 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:17:28.753765 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:17:33.984578 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:17:33.984546 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:17:33.985166 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:17:33.985147 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:17:33.987456 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:17:33.987435 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:17:33.988240 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:17:33.988205 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:17:33.993278 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:17:33.993258 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:17:33.993839 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:17:33.993818 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:17:40.792159 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:17:40.792127 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:18:01.855210 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:01.855171 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d65c755d-jvwcp"] Aug 04 00:18:01.858623 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:01.858601 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d65c755d-jvwcp" Aug 04 00:18:01.861018 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:01.860996 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zg8cz\"" Aug 04 00:18:01.861768 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:01.861752 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Aug 04 00:18:01.865759 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:01.865736 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d65c755d-jvwcp"] Aug 04 00:18:01.967921 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:01.967889 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/215639c2-cf65-4c3a-bc79-47bc79b10eb6-tls-cert\") pod \"authorino-7d65c755d-jvwcp\" (UID: \"215639c2-cf65-4c3a-bc79-47bc79b10eb6\") " pod="kuadrant-system/authorino-7d65c755d-jvwcp" Aug 04 00:18:01.968111 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:01.967974 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ggqr\" (UniqueName: \"kubernetes.io/projected/215639c2-cf65-4c3a-bc79-47bc79b10eb6-kube-api-access-6ggqr\") pod \"authorino-7d65c755d-jvwcp\" (UID: \"215639c2-cf65-4c3a-bc79-47bc79b10eb6\") " pod="kuadrant-system/authorino-7d65c755d-jvwcp" Aug 04 00:18:02.068464 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:02.068424 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/215639c2-cf65-4c3a-bc79-47bc79b10eb6-tls-cert\") pod \"authorino-7d65c755d-jvwcp\" (UID: \"215639c2-cf65-4c3a-bc79-47bc79b10eb6\") " pod="kuadrant-system/authorino-7d65c755d-jvwcp" Aug 04 00:18:02.068628 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:02.068497 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ggqr\" (UniqueName: \"kubernetes.io/projected/215639c2-cf65-4c3a-bc79-47bc79b10eb6-kube-api-access-6ggqr\") pod \"authorino-7d65c755d-jvwcp\" (UID: \"215639c2-cf65-4c3a-bc79-47bc79b10eb6\") " pod="kuadrant-system/authorino-7d65c755d-jvwcp" Aug 04 00:18:02.070879 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:02.070852 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/215639c2-cf65-4c3a-bc79-47bc79b10eb6-tls-cert\") pod \"authorino-7d65c755d-jvwcp\" (UID: \"215639c2-cf65-4c3a-bc79-47bc79b10eb6\") " pod="kuadrant-system/authorino-7d65c755d-jvwcp" Aug 04 00:18:02.076776 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:02.076744 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ggqr\" (UniqueName: \"kubernetes.io/projected/215639c2-cf65-4c3a-bc79-47bc79b10eb6-kube-api-access-6ggqr\") pod \"authorino-7d65c755d-jvwcp\" (UID: \"215639c2-cf65-4c3a-bc79-47bc79b10eb6\") " pod="kuadrant-system/authorino-7d65c755d-jvwcp" Aug 04 00:18:02.169703 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:02.169613 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d65c755d-jvwcp" Aug 04 00:18:02.306779 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:02.306754 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d65c755d-jvwcp"] Aug 04 00:18:02.309088 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:18:02.309052 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod215639c2_cf65_4c3a_bc79_47bc79b10eb6.slice/crio-0211157d02897d560f5f811e496b69409b913973fd7a12c047918d279911f628 WatchSource:0}: Error finding container 0211157d02897d560f5f811e496b69409b913973fd7a12c047918d279911f628: Status 404 returned error can't find the container with id 0211157d02897d560f5f811e496b69409b913973fd7a12c047918d279911f628 Aug 04 00:18:02.976052 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:02.976018 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d65c755d-jvwcp" event={"ID":"215639c2-cf65-4c3a-bc79-47bc79b10eb6","Type":"ContainerStarted","Data":"0211157d02897d560f5f811e496b69409b913973fd7a12c047918d279911f628"} Aug 04 00:18:03.984545 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:03.984505 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d65c755d-jvwcp" event={"ID":"215639c2-cf65-4c3a-bc79-47bc79b10eb6","Type":"ContainerStarted","Data":"50925284b40e2001a0a5a92d8f77743ab1d6ba8ff54aa2344708536e24a5d896"} Aug 04 00:18:04.001633 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:04.001572 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7d65c755d-jvwcp" podStartSLOduration=2.031429275 podStartE2EDuration="3.001551966s" totalImagesPullingTime="970.122691ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:18:03.999451163 +0000 UTC m=+930.482117493" watchObservedRunningTime="2026-08-04 00:18:04.001551966 +0000 UTC m=+930.484218335" Aug 04 00:18:04.531566 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:04.531528 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:18:04.608832 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:04.608791 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:18:05.677893 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:05.677839 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:18:05.775720 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:05.775648 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:18:16.132097 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:16.132047 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:18:16.183024 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:16.182992 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:18:33.431464 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:33.431416 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:18:43.116743 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:43.116704 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:18:54.532423 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:18:54.532381 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:20:11.531677 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:20:11.531638 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:20:12.806292 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:20:12.806257 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:20:36.034986 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:20:36.034892 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:20:45.432043 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:20:45.432006 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:21:15.797009 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:21:15.796970 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:21:15.896188 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:21:15.896145 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:21:16.035264 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:21:16.035216 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:21:21.693446 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:21:21.693408 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:21:21.799716 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:21:21.799678 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:21:21.904944 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:21:21.904908 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:21:22.037558 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:21:22.037516 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:21:24.033010 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:21:24.032975 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:21:27.778773 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:21:27.778718 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:21:27.880839 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:21:27.880801 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:21:43.430658 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:21:43.430616 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:21:50.631826 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:21:50.631784 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:22:06.125542 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:22:06.125445 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:22:12.027794 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:22:12.027755 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:22:21.439940 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:22:21.439877 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:22:28.831213 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:22:28.831174 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:22:34.016410 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:22:34.016383 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:22:34.018432 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:22:34.018398 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:22:34.026124 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:22:34.026094 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:22:34.027241 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:22:34.027202 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:22:34.037240 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:22:34.037204 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:22:34.038183 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:22:34.038162 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:22:42.221763 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:22:42.221724 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:22:47.232051 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:22:47.232002 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:22:52.227678 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:22:52.227639 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:22:59.134404 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:22:59.134367 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:23:04.921849 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:23:04.921803 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:23:17.595542 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:23:17.595490 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:23:28.020076 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:23:28.019976 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:23:50.638154 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:23:50.638110 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:24:01.025759 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:24:01.025712 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:24:07.531921 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:24:07.531879 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:24:15.632348 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:24:15.632302 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:24:26.426247 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:24:26.426201 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:24:28.169567 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:24:28.169527 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:24:32.118127 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:24:32.118089 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:24:40.016262 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:24:40.016207 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:24:51.537159 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:24:51.537123 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:24:58.922830 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:24:58.922742 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:25:09.527473 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:25:09.527427 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:25:14.632890 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:25:14.632851 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:25:20.926252 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:25:20.926200 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:25:29.729111 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:25:29.729071 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:25:35.336076 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:25:35.336036 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:25:45.727189 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:25:45.727151 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:26:49.635278 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:26:49.635174 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:27:32.881915 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:27:32.881877 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:27:32.981934 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:27:32.981900 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:27:34.059776 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:27:34.059740 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:27:34.062617 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:27:34.062594 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:27:34.062824 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:27:34.062799 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:27:34.065648 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:27:34.065627 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:27:34.068684 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:27:34.068666 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:27:34.071276 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:27:34.071243 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:27:54.565095 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:27:54.565043 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:27:54.665211 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:27:54.665173 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:28:02.019159 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:28:02.019074 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:28:07.130572 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:28:07.130530 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:28:31.531285 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:28:31.531249 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:28:42.725000 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:28:42.724962 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:28:53.223527 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:28:53.223486 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:28:58.426743 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:28:58.426699 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:29:03.424849 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:29:03.424807 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:29:13.326951 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:29:13.326911 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:29:18.532769 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:29:18.532733 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:29:28.332535 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:29:28.332454 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:29:33.523629 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:29:33.523586 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:29:43.527340 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:29:43.527304 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:29:50.227810 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:29:50.227759 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:29:59.330959 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:29:59.330924 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:30:21.227872 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:30:21.227832 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:30:27.826436 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:30:27.826389 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:30:30.977167 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:30:30.977132 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:30:32.823921 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:30:32.823882 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:30:51.355613 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:30:51.355562 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:30:54.135805 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:30:54.135755 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:31:27.646510 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:31:27.646472 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:31:29.216328 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:31:29.216289 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:31:29.532744 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:31:29.532700 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:31:39.624247 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:31:39.624190 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:31:43.924422 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:31:43.924380 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:31:50.918068 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:31:50.918018 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:32:01.241313 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:32:01.241271 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:32:07.626078 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:32:07.626041 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:32:14.829449 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:32:14.829407 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:32:23.118734 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:32:23.118694 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:32:27.255823 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:32:27.255732 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:32:33.024514 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:32:33.024476 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:32:34.097594 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:32:34.097555 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:32:34.100786 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:32:34.100761 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:32:34.103560 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:32:34.103531 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:32:34.106845 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:32:34.106819 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:32:34.108348 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:32:34.108325 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:32:34.113287 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:32:34.113266 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:32:35.451758 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:32:35.451703 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:32:39.531775 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:32:39.531734 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:32:55.723809 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:32:55.723768 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:33:02.527083 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:33:02.527044 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:33:14.228339 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:33:14.228301 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:33:20.125576 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:33:20.125539 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:33:27.831454 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:33:27.831417 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:33:31.569152 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:33:31.569123 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:33:33.130010 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:33:33.129966 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:33:40.025267 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:33:40.025232 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:33:44.523924 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:33:44.523878 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:33:50.927316 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:33:50.927277 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:34:01.525363 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:34:01.525286 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:34:07.540750 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:34:07.540710 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:34:18.428553 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:34:18.428515 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:34:33.720288 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:34:33.720249 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:34:39.535128 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:34:39.535086 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:34:43.288413 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:34:43.288381 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:34:45.823627 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:34:45.823590 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:34:57.534632 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:34:57.534592 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:35:05.822863 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:05.822822 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:35:09.167870 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:09.167838 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:11.027417 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:11.027377 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:35:12.115681 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:12.115648 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:12.217735 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:12.217656 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:12.557334 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:12.557278 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:18.100001 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:18.099963 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:18.169164 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:18.169116 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:18.322265 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:18.322191 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:18.634335 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:18.634306 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:18.801812 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:18.801776 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:20.791685 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:20.791638 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:20.891446 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:20.891411 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:21.919836 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:21.919789 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:22.020916 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:22.020882 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:23.493622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:23.493581 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:23.593778 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:23.593742 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:25.425156 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:25.425121 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:35:47.608580 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:47.608539 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:52.699783 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:52.699752 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:52.845059 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:52.845024 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:52.931067 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:52.931035 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:53.071255 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:53.071188 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:53.291270 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:53.291206 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:53.525924 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:53.525893 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:53.621241 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:53.621205 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:53.748530 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:53.748493 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:54.090050 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:54.090013 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:54.377584 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:54.377496 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:55.725994 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:55.725955 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:35:56.641444 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:56.641409 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:56.746821 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:56.746781 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:56.842396 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:56.842367 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:35:56.918937 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:35:56.918843 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:36:04.335905 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:04.335860 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:36:27.512467 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:27.512432 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:36:27.629696 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:27.629661 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:36:27.744207 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:27.744172 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:36:27.810935 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:27.810854 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:36:27.945412 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:27.945376 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:36:28.160524 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:28.160439 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:36:28.371452 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:28.371419 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:36:28.664119 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:28.664088 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:36:28.836448 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:28.836361 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:36:30.626336 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:30.626293 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:36:31.192955 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:31.192925 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:36:31.508456 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:31.508424 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:36:32.328439 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:32.328405 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:36:32.700868 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:32.700783 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:36:32.823385 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:32.823348 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:36:32.922392 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:32.922356 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:36:39.935413 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:39.935374 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:36:42.908275 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:42.908238 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:36:42.982687 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:36:42.982648 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:12.816686 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:12.816598 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:12.912083 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:12.912040 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:12.984545 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:12.984510 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:14.134107 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:14.134073 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:14.336903 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:14.336872 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:14.559837 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:14.559806 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:14.855324 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:14.855238 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:14.969729 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:14.969691 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:15.827872 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:15.827837 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:37:17.444950 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:17.444913 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:17.543558 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:17.543524 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:18.570141 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:18.570107 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:18.953601 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:18.953522 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:19.076688 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:19.076647 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:19.179407 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:19.179366 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:25.240116 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:25.240079 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:37:34.131406 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:34.131377 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:37:34.134256 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:34.134233 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:37:34.138680 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:34.138657 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:37:34.139736 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:34.139714 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:37:34.141570 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:34.141549 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:37:34.146909 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:34.146888 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:37:49.266581 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:49.266550 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:49.339136 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:49.339103 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:49.403634 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:49.403595 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:49.461585 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:49.461552 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:49.606624 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:49.606545 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:49.820304 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:49.820272 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:50.278663 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:50.278629 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:50.380755 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:50.380720 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:50.477964 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:50.477931 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:37:52.132775 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:37:52.132737 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:38:02.110894 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:38:02.110861 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:38:03.525473 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:38:03.525436 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:38:08.238368 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:38:08.238332 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:38:08.341393 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:38:08.341357 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:38:08.438626 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:38:08.438595 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:38:08.541055 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:38:08.540973 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:38:08.900911 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:38:08.900831 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:38:34.847892 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:38:34.847855 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:38:40.819473 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:38:40.819435 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:38:40.920266 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:38:40.920216 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:38:41.019480 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:38:41.019447 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:38:55.302192 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:38:55.302152 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:38:55.361236 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:38:55.361200 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:38:56.917352 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:38:56.917318 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:02.702617 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:02.702581 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:02.809759 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:02.809730 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:02.903448 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:02.903408 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:13.380021 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:13.379988 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:24.378791 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:24.378754 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:30.335549 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:30.335510 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:30.463514 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:30.463466 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:30.564811 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:30.564777 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:30.665899 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:30.665811 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:31.250276 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:31.250237 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:31.347789 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:31.347754 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:31.449420 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:31.449389 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:31.824813 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:31.824776 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:39:32.907801 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:32.907764 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:38.907854 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:38.907811 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:38.978736 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:38.978697 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:39.577250 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:39.577197 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:40.632599 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:40.632556 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:39:45.544037 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:45.543992 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:45.607300 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:45.607254 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:45.773273 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:45.773179 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:46.074452 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:46.074418 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:46.916563 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:46.916511 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:48.771707 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:48.771673 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:48.868206 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:48.868156 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:49.900652 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:49.900617 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:50.389040 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:50.389000 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:50.488656 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:50.488622 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:50.570935 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:50.570891 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:50.802829 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:50.802787 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:39:52.426756 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:52.426716 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:39:52.498880 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:39:52.498842 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:13.747157 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:13.747122 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:14.390919 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:14.390881 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:14.489080 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:14.489045 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:14.840020 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:14.839990 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:20.756007 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:20.755970 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:20.863980 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:20.863944 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:31.658794 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:31.658756 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:31.758052 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:31.758017 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:31.846322 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:31.846274 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:32.204428 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:32.204383 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:33.625992 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:33.625954 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:40:37.787931 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:37.787892 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:37.886474 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:37.886438 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:38.048042 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:38.047957 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:38.265470 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:38.265440 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:38.434018 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:38.433935 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:38.576842 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:38.576807 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:38.690192 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:38.690101 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:44.273132 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:44.273089 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:44.376154 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:44.376119 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:44.473643 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:44.473608 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:44.627174 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:44.627091 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:50.270810 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:50.270777 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:40:51.688188 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:40:51.688153 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:41:09.426606 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:41:09.426567 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:42:24.226128 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:42:24.226083 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:42:34.165628 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:42:34.165591 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:42:34.168540 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:42:34.168513 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:42:34.172799 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:42:34.172779 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:42:34.174055 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:42:34.174031 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:42:34.176105 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:42:34.176082 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:42:34.188043 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:42:34.188017 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:42:53.185683 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:42:53.185647 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:42:53.516294 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:42:53.516252 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:42:53.877593 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:42:53.877495 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:42:56.625800 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:42:56.625705 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:42:56.737310 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:42:56.737271 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:42:56.837446 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:42:56.837408 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:42:57.880058 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:42:57.880017 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:42:57.985997 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:42:57.985968 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:42:58.391476 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:42:58.391430 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:42:58.496361 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:42:58.496312 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:43:04.931607 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:43:04.931567 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:43:21.690632 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:43:21.690597 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:43:27.750391 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:43:27.750353 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:43:29.237422 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:43:29.237385 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:43:29.337742 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:43:29.337693 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:43:29.940858 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:43:29.940815 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:43:45.425935 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:43:45.425894 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:44:08.933873 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:44:08.933831 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:44:11.460186 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:44:11.460145 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:44:11.528419 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:44:11.528381 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:44:11.594086 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:44:11.594032 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:44:11.652870 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:44:11.652834 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:44:14.690359 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:44:14.690328 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:44:16.931841 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:44:16.931805 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:44:19.740027 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:44:19.739990 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:44:55.967888 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:44:55.967848 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:44:56.573773 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:44:56.573736 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:02.192940 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:02.192907 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:02.292949 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:02.292909 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:02.394035 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:02.393985 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:02.614506 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:02.614472 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:04.722533 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:04.722492 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:45:13.922979 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:13.922939 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:14.012150 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:14.012113 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:14.113506 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:14.113472 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:14.706861 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:14.706805 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:14.798472 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:14.798440 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:14.897771 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:14.897736 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:16.334031 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:16.333998 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:16.447601 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:16.447560 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:45:22.448717 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:22.448679 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:22.549189 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:22.549135 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:23.105657 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:23.105612 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:23.223681 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:23.223647 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:45:25.010836 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:25.010798 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:28.812597 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:28.812550 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:29.102904 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:29.102807 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:29.203661 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:29.203630 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:39.494302 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:39.494266 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:45:39.726611 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:45:39.726572 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:00.674588 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:00.674549 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:02.992900 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:02.992849 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:06.717462 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:06.717428 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:06.782390 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:06.782353 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:07.063576 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:07.063536 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:07.375701 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:07.375616 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:07.844797 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:07.844762 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:10.707846 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:10.707799 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:10.810284 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:10.810211 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:11.936916 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:11.936874 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:12.133619 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:12.133578 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:12.496290 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:12.496255 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:12.600353 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:12.600320 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:14.520749 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:14.520709 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:46:35.684840 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:35.684797 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:36.022930 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:36.022899 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:36.348735 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:36.348645 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:36.893058 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:36.893026 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:38.845529 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:38.845494 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:38.942635 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:38.942596 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:39.982842 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:39.982808 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:40.474839 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:40.474800 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:40.570824 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:40.570782 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:40.644563 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:40.644530 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:46:42.729003 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:46:42.728967 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:47:03.788419 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:03.788381 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:47:07.461511 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:07.461455 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:47:10.020841 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:10.020795 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:47:17.930809 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:17.930765 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:47:33.925368 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:33.925287 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv"] Aug 04 00:47:33.928869 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:33.928851 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:33.932289 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:33.932257 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-743d5d\"/\"kube-root-ca.crt\"" Aug 04 00:47:33.932438 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:33.932294 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-743d5d\"/\"test-model-743d5d-kserve-self-signed-certs\"" Aug 04 00:47:33.932438 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:33.932308 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-743d5d\"/\"default-dockercfg-5ktp6\"" Aug 04 00:47:33.932438 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:33.932298 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-743d5d\"/\"openshift-service-ca.crt\"" Aug 04 00:47:33.939513 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:33.939485 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv"] Aug 04 00:47:33.949867 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:33.949829 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:47:34.000535 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.000497 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttwxr\" (UniqueName: \"kubernetes.io/projected/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-kube-api-access-ttwxr\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.000704 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.000550 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-dshm\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.000704 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.000631 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-model-cache\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.000704 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.000697 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-tmp-dir\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.000811 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.000742 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-home\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.000811 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.000759 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-tls-certs\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.101681 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.101646 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-tls-certs\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.101914 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.101704 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttwxr\" (UniqueName: \"kubernetes.io/projected/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-kube-api-access-ttwxr\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.101914 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.101735 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-dshm\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.101914 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.101770 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-model-cache\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.102175 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.101916 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-tmp-dir\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.102175 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.101967 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-home\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.102311 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.102279 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-model-cache\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.102427 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.102333 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-home\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.102427 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.102357 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-tmp-dir\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.104084 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.104066 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-dshm\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.104290 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.104274 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-tls-certs\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.112284 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.112261 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttwxr\" (UniqueName: \"kubernetes.io/projected/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-kube-api-access-ttwxr\") pod \"test-model-743d5d-kserve-5dd59dbdf9-2l6pv\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.203731 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.203655 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:47:34.206499 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.206475 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:47:34.212006 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.211983 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:47:34.212154 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.212092 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:47:34.214576 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.214556 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:47:34.219770 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.219752 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:47:34.239045 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.238999 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:34.380698 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.380670 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv"] Aug 04 00:47:34.383376 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:47:34.383345 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa43d63c_a8cb_4f4d_978d_2c7e5f4ab877.slice/crio-f5496aecc3490762ed4e69ac8a3af30df5b653dd2c70ae22ed4dc92d3de4515d WatchSource:0}: Error finding container f5496aecc3490762ed4e69ac8a3af30df5b653dd2c70ae22ed4dc92d3de4515d: Status 404 returned error can't find the container with id f5496aecc3490762ed4e69ac8a3af30df5b653dd2c70ae22ed4dc92d3de4515d Aug 04 00:47:34.713217 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.713173 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" event={"ID":"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877","Type":"ContainerStarted","Data":"ebbc1ab7bb441af99e373139146f93e4d1fba25e95665f66f7c9d8112082be33"} Aug 04 00:47:34.713217 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:34.713217 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" event={"ID":"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877","Type":"ContainerStarted","Data":"f5496aecc3490762ed4e69ac8a3af30df5b653dd2c70ae22ed4dc92d3de4515d"} Aug 04 00:47:37.926237 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:37.926161 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" podStartSLOduration=4.926138116 podStartE2EDuration="4.926138116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:47:34.728799063 +0000 UTC m=+2701.211465394" watchObservedRunningTime="2026-08-04 00:47:37.926138116 +0000 UTC m=+2704.408804447" Aug 04 00:47:37.928083 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:37.928053 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:47:44.239178 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:44.239134 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:44.239900 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:44.239197 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:44.251388 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:44.251361 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:44.772099 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:44.772069 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:47:54.718437 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:54.718400 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:47:54.854948 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:47:54.854901 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:13.032270 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:13.032214 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:14.035725 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:14.035696 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv"] Aug 04 00:48:14.046742 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:14.046706 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:16.425205 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:16.425170 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:48:18.788430 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:18.788392 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:18.887663 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:18.887624 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:18.973180 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:18.973149 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:19.038742 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:19.038640 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:19.334917 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:19.334836 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:24.409846 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:24.409808 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:24.521321 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:24.521279 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:48:29.056059 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.055987 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" podUID="fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877" containerName="main" containerID="cri-o://ebbc1ab7bb441af99e373139146f93e4d1fba25e95665f66f7c9d8112082be33" gracePeriod=45 Aug 04 00:48:29.314528 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.314449 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:48:29.432779 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.432736 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-tmp-dir\") pod \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " Aug 04 00:48:29.432960 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.432801 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttwxr\" (UniqueName: \"kubernetes.io/projected/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-kube-api-access-ttwxr\") pod \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " Aug 04 00:48:29.432960 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.432836 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-home\") pod \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " Aug 04 00:48:29.432960 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.432893 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-dshm\") pod \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " Aug 04 00:48:29.432960 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.432925 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-tls-certs\") pod \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " Aug 04 00:48:29.433174 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.432991 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-model-cache\") pod \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\" (UID: \"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877\") " Aug 04 00:48:29.433174 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.433036 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877" (UID: "fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:29.433174 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.433128 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-home" (OuterVolumeSpecName: "home") pod "fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877" (UID: "fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:29.433527 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.433270 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877" (UID: "fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:29.433527 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.433341 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:48:29.433527 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.433360 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:48:29.435163 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.435140 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-dshm" (OuterVolumeSpecName: "dshm") pod "fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877" (UID: "fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:48:29.435163 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.435144 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-kube-api-access-ttwxr" (OuterVolumeSpecName: "kube-api-access-ttwxr") pod "fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877" (UID: "fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877"). InnerVolumeSpecName "kube-api-access-ttwxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:48:29.435325 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.435172 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877" (UID: "fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:48:29.534848 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.534805 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:48:29.534848 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.534838 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:48:29.534848 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.534849 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:48:29.534848 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.534858 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttwxr\" (UniqueName: \"kubernetes.io/projected/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877-kube-api-access-ttwxr\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:48:29.986719 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.986682 2570 generic.go:356] "Generic (PLEG): container finished" podID="fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877" containerID="ebbc1ab7bb441af99e373139146f93e4d1fba25e95665f66f7c9d8112082be33" exitCode=0 Aug 04 00:48:29.986719 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.986725 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" event={"ID":"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877","Type":"ContainerDied","Data":"ebbc1ab7bb441af99e373139146f93e4d1fba25e95665f66f7c9d8112082be33"} Aug 04 00:48:29.986987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.986728 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" Aug 04 00:48:29.986987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.986754 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv" event={"ID":"fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877","Type":"ContainerDied","Data":"f5496aecc3490762ed4e69ac8a3af30df5b653dd2c70ae22ed4dc92d3de4515d"} Aug 04 00:48:29.986987 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.986771 2570 scope.go:117] "RemoveContainer" containerID="ebbc1ab7bb441af99e373139146f93e4d1fba25e95665f66f7c9d8112082be33" Aug 04 00:48:29.996045 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.996021 2570 scope.go:117] "RemoveContainer" containerID="ebbc1ab7bb441af99e373139146f93e4d1fba25e95665f66f7c9d8112082be33" Aug 04 00:48:29.996349 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:48:29.996324 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebbc1ab7bb441af99e373139146f93e4d1fba25e95665f66f7c9d8112082be33\": container with ID starting with ebbc1ab7bb441af99e373139146f93e4d1fba25e95665f66f7c9d8112082be33 not found: ID does not exist" containerID="ebbc1ab7bb441af99e373139146f93e4d1fba25e95665f66f7c9d8112082be33" Aug 04 00:48:29.996430 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:29.996362 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbc1ab7bb441af99e373139146f93e4d1fba25e95665f66f7c9d8112082be33"} err="failed to get container status \"ebbc1ab7bb441af99e373139146f93e4d1fba25e95665f66f7c9d8112082be33\": rpc error: code = NotFound desc = could not find container \"ebbc1ab7bb441af99e373139146f93e4d1fba25e95665f66f7c9d8112082be33\": container with ID starting with ebbc1ab7bb441af99e373139146f93e4d1fba25e95665f66f7c9d8112082be33 not found: ID does not exist" Aug 04 00:48:30.006473 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:30.006437 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv"] Aug 04 00:48:30.010677 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:30.010648 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-743d5d/test-model-743d5d-kserve-5dd59dbdf9-2l6pv"] Aug 04 00:48:30.033778 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:30.033744 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877" path="/var/lib/kubelet/pods/fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877/volumes" Aug 04 00:48:30.183401 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:30.183356 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:30.427544 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:30.427440 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:30.644175 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:30.644137 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:41.209062 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:41.209029 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:41.311109 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:41.311068 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:41.374378 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:41.374338 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:41.734337 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:41.734299 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:43.527280 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:43.527249 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:48:47.655606 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:47.655566 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:47.720582 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:47.720544 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:47.937729 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:47.937631 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:48.204631 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:48.204546 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:48.565280 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:48.565246 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:48.860505 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:48.860415 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:49.233932 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:49.233824 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:51.577986 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:51.577948 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:51.683355 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:51.683321 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:52.763655 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:52.763618 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:52.902018 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:52.901980 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:53.241616 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:53.241579 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:53.362196 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:53.362156 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:48:54.823629 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:48:54.823592 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:49:16.526531 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:49:16.526484 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:49:16.961404 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:49:16.961294 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:49:17.294948 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:49:17.294902 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:49:17.393746 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:49:17.393703 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:49:19.696424 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:49:19.696380 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:49:19.843577 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:49:19.843537 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:49:20.821393 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:49:20.821358 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:49:21.316447 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:49:21.316409 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:49:21.412771 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:49:21.412735 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:49:21.611340 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:49:21.611242 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:49:24.123007 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:49:24.122960 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:49:44.595263 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:49:44.595208 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:49:52.680797 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:49:52.680757 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:02.937778 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:02.937735 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:03.008734 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:03.008680 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:03.068123 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:03.068072 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:03.199307 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:03.199196 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:03.586332 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:03.586290 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:05.226087 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:05.226051 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:50:09.248575 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:09.248541 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:09.312100 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:09.312059 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:09.407264 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:09.407212 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:09.481031 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:09.480997 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:09.547353 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:09.547268 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:09.722748 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:09.722712 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:25.312529 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:25.312490 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:25.412965 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:25.412936 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:25.537241 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:25.537199 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:25.623685 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:25.623598 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:25.895072 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:25.894987 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:27.326787 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:27.326752 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:50:31.698146 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:31.698105 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:31.799074 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:31.799029 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:31.948208 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:31.948120 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:32.178724 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:32.178691 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:32.520024 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:32.519987 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:32.841153 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:32.841067 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:33.126790 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:33.126710 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:36.080151 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:36.080108 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:36.180130 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:36.180101 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:37.214433 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:37.214405 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:37.324731 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:37.324683 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:37.721489 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:37.721457 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:37.821282 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:37.821218 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:40.834750 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:40.834719 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:50:50.801362 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:50.801319 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:51.106599 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:51.106528 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:51.413700 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:51.413609 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:51.816545 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:51.816504 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:54.328429 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:54.328399 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:54.422693 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:54.422655 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:55.940459 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:55.940425 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:56.230769 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:56.230676 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:56.439099 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:56.439067 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:56.539247 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:56.539199 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:50:59.022403 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:50:59.022369 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:51:08.947577 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:08.947536 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:51:25.221030 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.220997 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr"] Aug 04 00:51:25.221590 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.221571 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877" containerName="main" Aug 04 00:51:25.221677 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.221593 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877" containerName="main" Aug 04 00:51:25.221729 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.221695 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa43d63c-a8cb-4f4d-978d-2c7e5f4ab877" containerName="main" Aug 04 00:51:25.224626 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.224604 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.228139 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.228116 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5ddf6e\"/\"auth-test-model-5ddf6e-kserve-self-signed-certs\"" Aug 04 00:51:25.228280 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.228151 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5ddf6e\"/\"kube-root-ca.crt\"" Aug 04 00:51:25.228280 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.228201 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5ddf6e\"/\"default-dockercfg-p97hc\"" Aug 04 00:51:25.228280 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.228188 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5ddf6e\"/\"openshift-service-ca.crt\"" Aug 04 00:51:25.233359 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.233339 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr"] Aug 04 00:51:25.251113 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.251087 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:51:25.370701 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.370666 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-home\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.370701 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.370702 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-tmp-dir\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.370895 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.370778 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-dshm\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.370895 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.370819 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-model-cache\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.370895 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.370872 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk447\" (UniqueName: \"kubernetes.io/projected/10a112fc-4608-4bd0-b75a-5bb392702c6f-kube-api-access-hk447\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.371008 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.370906 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a112fc-4608-4bd0-b75a-5bb392702c6f-tls-certs\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.471990 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.471914 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk447\" (UniqueName: \"kubernetes.io/projected/10a112fc-4608-4bd0-b75a-5bb392702c6f-kube-api-access-hk447\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.471990 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.471956 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a112fc-4608-4bd0-b75a-5bb392702c6f-tls-certs\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.471990 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.471990 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-home\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.472248 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.472011 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-tmp-dir\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.472248 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.472058 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-dshm\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.472248 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.472081 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-model-cache\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.472448 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.472422 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-home\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.472572 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.472511 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-tmp-dir\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.472572 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.472556 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-model-cache\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.474443 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.474422 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-dshm\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.474606 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.474589 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a112fc-4608-4bd0-b75a-5bb392702c6f-tls-certs\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.481161 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.481139 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk447\" (UniqueName: \"kubernetes.io/projected/10a112fc-4608-4bd0-b75a-5bb392702c6f-kube-api-access-hk447\") pod \"auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.534840 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.534819 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:25.662374 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.662349 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr"] Aug 04 00:51:25.664313 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:51:25.664278 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10a112fc_4608_4bd0_b75a_5bb392702c6f.slice/crio-2d304c574f0ddbc8730b3a79e24f97ef0711977305380d85d7c8801e53cb1db4 WatchSource:0}: Error finding container 2d304c574f0ddbc8730b3a79e24f97ef0711977305380d85d7c8801e53cb1db4: Status 404 returned error can't find the container with id 2d304c574f0ddbc8730b3a79e24f97ef0711977305380d85d7c8801e53cb1db4 Aug 04 00:51:25.844238 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.844190 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" event={"ID":"10a112fc-4608-4bd0-b75a-5bb392702c6f","Type":"ContainerStarted","Data":"9ad2d566d5b67383fe3bac474ee13d50ef980cca9e112816f8348de448d8a376"} Aug 04 00:51:25.844372 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.844264 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" event={"ID":"10a112fc-4608-4bd0-b75a-5bb392702c6f","Type":"ContainerStarted","Data":"2d304c574f0ddbc8730b3a79e24f97ef0711977305380d85d7c8801e53cb1db4"} Aug 04 00:51:25.860528 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:25.860481 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" podStartSLOduration=0.86046787 podStartE2EDuration="860.46787ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:51:25.859589887 +0000 UTC m=+2932.342256218" watchObservedRunningTime="2026-08-04 00:51:25.86046787 +0000 UTC m=+2932.343134198" Aug 04 00:51:27.528690 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:27.528648 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:51:35.534997 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:35.534962 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:35.534997 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:35.535003 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:35.547514 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:35.547489 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:35.902300 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:35.902203 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:51:37.656121 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:37.656086 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:51:46.748949 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:46.748914 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:51:49.131212 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:51:49.131176 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:52:19.424351 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:52:19.424317 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:52:34.234624 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:52:34.234591 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:52:34.237508 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:52:34.237483 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:52:34.242749 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:52:34.242731 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:52:34.243502 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:52:34.243485 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:52:34.246273 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:52:34.246253 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:52:34.258060 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:52:34.258038 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:52:36.220793 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:52:36.220758 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:52:46.740147 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:52:46.740111 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:52:56.479145 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:52:56.479107 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:52:56.576530 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:52:56.576507 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:53:02.119966 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:53:02.119932 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:53:17.619177 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:53:17.619139 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:53:28.732606 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:53:28.732528 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:53:44.231819 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:53:44.231783 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:53:59.344381 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:53:59.344346 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:54:10.123648 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:54:10.123617 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:54:25.928283 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:54:25.928248 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:54:39.032948 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:54:39.032916 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:54:39.117604 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:54:39.117573 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:54:41.627000 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:54:41.626965 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:54:50.021274 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:54:50.021235 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:54:53.220467 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:54:53.220436 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:54:53.320232 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:54:53.320198 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:54:58.636929 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:54:58.636833 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:55:08.830917 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:55:08.830876 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:55:20.132121 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:55:20.132086 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:55:35.022340 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:55:35.022298 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:55:51.819348 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:55:51.819313 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:55:55.888251 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:55:55.888205 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr"] Aug 04 00:55:55.912671 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:55:55.912643 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:55:56.861091 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:55:56.861059 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:56:02.323345 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:02.323312 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:56:10.907191 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:10.907150 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" podUID="10a112fc-4608-4bd0-b75a-5bb392702c6f" containerName="main" containerID="cri-o://9ad2d566d5b67383fe3bac474ee13d50ef980cca9e112816f8348de448d8a376" gracePeriod=45 Aug 04 00:56:11.149144 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.149121 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:56:11.152083 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.152065 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-home\") pod \"10a112fc-4608-4bd0-b75a-5bb392702c6f\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " Aug 04 00:56:11.152195 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.152095 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-model-cache\") pod \"10a112fc-4608-4bd0-b75a-5bb392702c6f\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " Aug 04 00:56:11.152195 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.152157 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hk447\" (UniqueName: \"kubernetes.io/projected/10a112fc-4608-4bd0-b75a-5bb392702c6f-kube-api-access-hk447\") pod \"10a112fc-4608-4bd0-b75a-5bb392702c6f\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " Aug 04 00:56:11.152334 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.152196 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a112fc-4608-4bd0-b75a-5bb392702c6f-tls-certs\") pod \"10a112fc-4608-4bd0-b75a-5bb392702c6f\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " Aug 04 00:56:11.152334 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.152272 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-dshm\") pod \"10a112fc-4608-4bd0-b75a-5bb392702c6f\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " Aug 04 00:56:11.152441 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.152332 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-tmp-dir\") pod \"10a112fc-4608-4bd0-b75a-5bb392702c6f\" (UID: \"10a112fc-4608-4bd0-b75a-5bb392702c6f\") " Aug 04 00:56:11.152441 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.152331 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-home" (OuterVolumeSpecName: "home") pod "10a112fc-4608-4bd0-b75a-5bb392702c6f" (UID: "10a112fc-4608-4bd0-b75a-5bb392702c6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:56:11.152531 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.152446 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-model-cache" (OuterVolumeSpecName: "model-cache") pod "10a112fc-4608-4bd0-b75a-5bb392702c6f" (UID: "10a112fc-4608-4bd0-b75a-5bb392702c6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:56:11.152689 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.152664 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:56:11.152798 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.152696 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:56:11.152798 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.152668 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10a112fc-4608-4bd0-b75a-5bb392702c6f" (UID: "10a112fc-4608-4bd0-b75a-5bb392702c6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:56:11.154364 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.154344 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a112fc-4608-4bd0-b75a-5bb392702c6f-kube-api-access-hk447" (OuterVolumeSpecName: "kube-api-access-hk447") pod "10a112fc-4608-4bd0-b75a-5bb392702c6f" (UID: "10a112fc-4608-4bd0-b75a-5bb392702c6f"). InnerVolumeSpecName "kube-api-access-hk447". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:56:11.154721 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.154680 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a112fc-4608-4bd0-b75a-5bb392702c6f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10a112fc-4608-4bd0-b75a-5bb392702c6f" (UID: "10a112fc-4608-4bd0-b75a-5bb392702c6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:56:11.154721 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.154693 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-dshm" (OuterVolumeSpecName: "dshm") pod "10a112fc-4608-4bd0-b75a-5bb392702c6f" (UID: "10a112fc-4608-4bd0-b75a-5bb392702c6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:56:11.208895 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.208868 2570 generic.go:356] "Generic (PLEG): container finished" podID="10a112fc-4608-4bd0-b75a-5bb392702c6f" containerID="9ad2d566d5b67383fe3bac474ee13d50ef980cca9e112816f8348de448d8a376" exitCode=0 Aug 04 00:56:11.209011 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.208909 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" Aug 04 00:56:11.209011 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.208928 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" event={"ID":"10a112fc-4608-4bd0-b75a-5bb392702c6f","Type":"ContainerDied","Data":"9ad2d566d5b67383fe3bac474ee13d50ef980cca9e112816f8348de448d8a376"} Aug 04 00:56:11.209011 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.208957 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr" event={"ID":"10a112fc-4608-4bd0-b75a-5bb392702c6f","Type":"ContainerDied","Data":"2d304c574f0ddbc8730b3a79e24f97ef0711977305380d85d7c8801e53cb1db4"} Aug 04 00:56:11.209011 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.208974 2570 scope.go:117] "RemoveContainer" containerID="9ad2d566d5b67383fe3bac474ee13d50ef980cca9e112816f8348de448d8a376" Aug 04 00:56:11.217551 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.217534 2570 scope.go:117] "RemoveContainer" containerID="9ad2d566d5b67383fe3bac474ee13d50ef980cca9e112816f8348de448d8a376" Aug 04 00:56:11.217819 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:56:11.217801 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad2d566d5b67383fe3bac474ee13d50ef980cca9e112816f8348de448d8a376\": container with ID starting with 9ad2d566d5b67383fe3bac474ee13d50ef980cca9e112816f8348de448d8a376 not found: ID does not exist" containerID="9ad2d566d5b67383fe3bac474ee13d50ef980cca9e112816f8348de448d8a376" Aug 04 00:56:11.217877 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.217829 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad2d566d5b67383fe3bac474ee13d50ef980cca9e112816f8348de448d8a376"} err="failed to get container status \"9ad2d566d5b67383fe3bac474ee13d50ef980cca9e112816f8348de448d8a376\": rpc error: code = NotFound desc = could not find container \"9ad2d566d5b67383fe3bac474ee13d50ef980cca9e112816f8348de448d8a376\": container with ID starting with 9ad2d566d5b67383fe3bac474ee13d50ef980cca9e112816f8348de448d8a376 not found: ID does not exist" Aug 04 00:56:11.229287 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.229265 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr"] Aug 04 00:56:11.233261 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.233242 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5ddf6e/auth-test-model-5ddf6e-kserve-75b7cff48c-wmnwr"] Aug 04 00:56:11.253826 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.253801 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:56:11.253826 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.253822 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk447\" (UniqueName: \"kubernetes.io/projected/10a112fc-4608-4bd0-b75a-5bb392702c6f-kube-api-access-hk447\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:56:11.253937 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.253832 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a112fc-4608-4bd0-b75a-5bb392702c6f-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:56:11.253937 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:11.253841 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a112fc-4608-4bd0-b75a-5bb392702c6f-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:56:12.034132 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:12.034092 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10a112fc-4608-4bd0-b75a-5bb392702c6f" path="/var/lib/kubelet/pods/10a112fc-4608-4bd0-b75a-5bb392702c6f/volumes" Aug 04 00:56:17.817756 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:17.817696 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:56:19.531275 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:19.531239 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:56:32.010998 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:32.010917 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:56:32.065694 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:32.065669 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:56:36.624547 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.624507 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt"] Aug 04 00:56:36.625013 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.624884 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a112fc-4608-4bd0-b75a-5bb392702c6f" containerName="main" Aug 04 00:56:36.625013 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.624903 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a112fc-4608-4bd0-b75a-5bb392702c6f" containerName="main" Aug 04 00:56:36.625150 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.625029 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a112fc-4608-4bd0-b75a-5bb392702c6f" containerName="main" Aug 04 00:56:36.629547 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.629520 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.631855 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.631828 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-33bda8\"/\"openshift-service-ca.crt\"" Aug 04 00:56:36.632793 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.632771 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-33bda8\"/\"kube-root-ca.crt\"" Aug 04 00:56:36.632901 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.632773 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-33bda8\"/\"sub-test-model-33bda8-kserve-self-signed-certs\"" Aug 04 00:56:36.632901 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.632773 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-33bda8\"/\"default-dockercfg-5qnwp\"" Aug 04 00:56:36.636796 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.636762 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt"] Aug 04 00:56:36.656779 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.656751 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:56:36.751622 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.751595 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-home\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.751761 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.751632 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-dshm\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.751761 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.751659 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-model-cache\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.751855 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.751757 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-tls-certs\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.751855 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.751805 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwz2k\" (UniqueName: \"kubernetes.io/projected/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-kube-api-access-jwz2k\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.751956 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.751891 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-tmp-dir\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.852656 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.852630 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-tmp-dir\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.852756 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.852676 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-home\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.852756 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.852691 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-dshm\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.852901 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.852874 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-model-cache\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.853001 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.852941 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-tls-certs\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.853068 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.852998 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwz2k\" (UniqueName: \"kubernetes.io/projected/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-kube-api-access-jwz2k\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.853127 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.853082 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-home\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.853127 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.853118 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-tmp-dir\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.853259 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.853148 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-model-cache\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.854820 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.854800 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-dshm\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.855209 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.855191 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-tls-certs\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.864340 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.864320 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwz2k\" (UniqueName: \"kubernetes.io/projected/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-kube-api-access-jwz2k\") pod \"sub-test-model-33bda8-kserve-7766bf7ddf-7czzt\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:36.940181 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:36.940126 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:37.071197 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:37.071175 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt"] Aug 04 00:56:37.072794 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:56:37.072771 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddccd11ee_e117_45df_9354_8a0cbd6dbaf0.slice/crio-0fb437f7b80947094677b105b48c8cd915cf7785a624b2bfb3b1e01de34ed8cd WatchSource:0}: Error finding container 0fb437f7b80947094677b105b48c8cd915cf7785a624b2bfb3b1e01de34ed8cd: Status 404 returned error can't find the container with id 0fb437f7b80947094677b105b48c8cd915cf7785a624b2bfb3b1e01de34ed8cd Aug 04 00:56:37.334601 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:37.334569 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" event={"ID":"dccd11ee-e117-45df-9354-8a0cbd6dbaf0","Type":"ContainerStarted","Data":"01eb49ddf2db6b2c3e2204fba4b9dbc4aa73a63da1895ac981fecacc1f0e696e"} Aug 04 00:56:37.334601 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:37.334612 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" event={"ID":"dccd11ee-e117-45df-9354-8a0cbd6dbaf0","Type":"ContainerStarted","Data":"0fb437f7b80947094677b105b48c8cd915cf7785a624b2bfb3b1e01de34ed8cd"} Aug 04 00:56:37.352885 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:37.352844 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" podStartSLOduration=1.352828883 podStartE2EDuration="1.352828883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:56:37.351684379 +0000 UTC m=+3243.834350709" watchObservedRunningTime="2026-08-04 00:56:37.352828883 +0000 UTC m=+3243.835495232" Aug 04 00:56:39.327937 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:39.327896 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:56:46.940514 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:46.940477 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:46.940885 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:46.940552 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:46.952864 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:46.952843 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:56:47.406142 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:56:47.406108 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:57:14.432890 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:57:14.432848 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:57:25.524583 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:57:25.524547 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:57:34.265473 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:57:34.265444 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:57:34.268061 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:57:34.268041 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:57:34.273902 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:57:34.273884 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:57:34.283216 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:57:34.283199 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 00:57:34.285723 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:57:34.285707 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 00:57:34.290961 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:57:34.290946 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 00:57:53.227426 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:57:53.227380 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:57:57.637641 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:57:57.637561 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:57:58.520657 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:57:58.520619 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt"] Aug 04 00:57:58.539826 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:57:58.539791 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:58:03.624749 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:03.624708 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:58:10.591707 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:10.591649 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:58:12.255699 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:12.255662 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:58:13.540928 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.540891 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" podUID="dccd11ee-e117-45df-9354-8a0cbd6dbaf0" containerName="main" containerID="cri-o://01eb49ddf2db6b2c3e2204fba4b9dbc4aa73a63da1895ac981fecacc1f0e696e" gracePeriod=45 Aug 04 00:58:13.780341 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.780320 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:58:13.824581 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.824505 2570 generic.go:356] "Generic (PLEG): container finished" podID="dccd11ee-e117-45df-9354-8a0cbd6dbaf0" containerID="01eb49ddf2db6b2c3e2204fba4b9dbc4aa73a63da1895ac981fecacc1f0e696e" exitCode=0 Aug 04 00:58:13.824581 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.824535 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" Aug 04 00:58:13.824581 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.824552 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" event={"ID":"dccd11ee-e117-45df-9354-8a0cbd6dbaf0","Type":"ContainerDied","Data":"01eb49ddf2db6b2c3e2204fba4b9dbc4aa73a63da1895ac981fecacc1f0e696e"} Aug 04 00:58:13.824831 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.824592 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt" event={"ID":"dccd11ee-e117-45df-9354-8a0cbd6dbaf0","Type":"ContainerDied","Data":"0fb437f7b80947094677b105b48c8cd915cf7785a624b2bfb3b1e01de34ed8cd"} Aug 04 00:58:13.824831 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.824615 2570 scope.go:117] "RemoveContainer" containerID="01eb49ddf2db6b2c3e2204fba4b9dbc4aa73a63da1895ac981fecacc1f0e696e" Aug 04 00:58:13.832713 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.832697 2570 scope.go:117] "RemoveContainer" containerID="01eb49ddf2db6b2c3e2204fba4b9dbc4aa73a63da1895ac981fecacc1f0e696e" Aug 04 00:58:13.832955 ip-10-0-138-102 kubenswrapper[2570]: E0804 00:58:13.832934 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01eb49ddf2db6b2c3e2204fba4b9dbc4aa73a63da1895ac981fecacc1f0e696e\": container with ID starting with 01eb49ddf2db6b2c3e2204fba4b9dbc4aa73a63da1895ac981fecacc1f0e696e not found: ID does not exist" containerID="01eb49ddf2db6b2c3e2204fba4b9dbc4aa73a63da1895ac981fecacc1f0e696e" Aug 04 00:58:13.833026 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.832966 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01eb49ddf2db6b2c3e2204fba4b9dbc4aa73a63da1895ac981fecacc1f0e696e"} err="failed to get container status \"01eb49ddf2db6b2c3e2204fba4b9dbc4aa73a63da1895ac981fecacc1f0e696e\": rpc error: code = NotFound desc = could not find container \"01eb49ddf2db6b2c3e2204fba4b9dbc4aa73a63da1895ac981fecacc1f0e696e\": container with ID starting with 01eb49ddf2db6b2c3e2204fba4b9dbc4aa73a63da1895ac981fecacc1f0e696e not found: ID does not exist" Aug 04 00:58:13.840279 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.840248 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-tmp-dir\") pod \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " Aug 04 00:58:13.840366 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.840311 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-tls-certs\") pod \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " Aug 04 00:58:13.840366 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.840351 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-home\") pod \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " Aug 04 00:58:13.840461 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.840366 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-model-cache\") pod \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " Aug 04 00:58:13.840461 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.840397 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwz2k\" (UniqueName: \"kubernetes.io/projected/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-kube-api-access-jwz2k\") pod \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " Aug 04 00:58:13.840461 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.840422 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-dshm\") pod \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\" (UID: \"dccd11ee-e117-45df-9354-8a0cbd6dbaf0\") " Aug 04 00:58:13.840461 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.840447 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dccd11ee-e117-45df-9354-8a0cbd6dbaf0" (UID: "dccd11ee-e117-45df-9354-8a0cbd6dbaf0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:58:13.840668 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.840584 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-home" (OuterVolumeSpecName: "home") pod "dccd11ee-e117-45df-9354-8a0cbd6dbaf0" (UID: "dccd11ee-e117-45df-9354-8a0cbd6dbaf0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:58:13.840668 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.840620 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-model-cache" (OuterVolumeSpecName: "model-cache") pod "dccd11ee-e117-45df-9354-8a0cbd6dbaf0" (UID: "dccd11ee-e117-45df-9354-8a0cbd6dbaf0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:58:13.840830 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.840812 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:58:13.840882 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.840835 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:58:13.840882 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.840848 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:58:13.842525 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.842502 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-kube-api-access-jwz2k" (OuterVolumeSpecName: "kube-api-access-jwz2k") pod "dccd11ee-e117-45df-9354-8a0cbd6dbaf0" (UID: "dccd11ee-e117-45df-9354-8a0cbd6dbaf0"). InnerVolumeSpecName "kube-api-access-jwz2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:58:13.842636 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.842586 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dccd11ee-e117-45df-9354-8a0cbd6dbaf0" (UID: "dccd11ee-e117-45df-9354-8a0cbd6dbaf0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:58:13.842709 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.842669 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-dshm" (OuterVolumeSpecName: "dshm") pod "dccd11ee-e117-45df-9354-8a0cbd6dbaf0" (UID: "dccd11ee-e117-45df-9354-8a0cbd6dbaf0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:58:13.941675 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.941652 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:58:13.941675 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.941673 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwz2k\" (UniqueName: \"kubernetes.io/projected/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-kube-api-access-jwz2k\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:58:13.941831 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:13.941682 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccd11ee-e117-45df-9354-8a0cbd6dbaf0-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 00:58:14.139534 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:14.139461 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt"] Aug 04 00:58:14.144737 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:14.144717 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/sub-test-model-33bda8-kserve-7766bf7ddf-7czzt"] Aug 04 00:58:14.628021 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:14.627989 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:58:16.034811 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:16.034775 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dccd11ee-e117-45df-9354-8a0cbd6dbaf0" path="/var/lib/kubelet/pods/dccd11ee-e117-45df-9354-8a0cbd6dbaf0/volumes" Aug 04 00:58:35.618561 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.618526 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh"] Aug 04 00:58:35.619035 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.618989 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dccd11ee-e117-45df-9354-8a0cbd6dbaf0" containerName="main" Aug 04 00:58:35.619035 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.619006 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dccd11ee-e117-45df-9354-8a0cbd6dbaf0" containerName="main" Aug 04 00:58:35.619122 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.619112 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="dccd11ee-e117-45df-9354-8a0cbd6dbaf0" containerName="main" Aug 04 00:58:35.623360 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.623344 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.626982 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.626962 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-33bda8\"/\"default-dockercfg-5qnwp\"" Aug 04 00:58:35.627114 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.626978 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-33bda8\"/\"rate-test-model-33bda8-kserve-self-signed-certs\"" Aug 04 00:58:35.627114 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.627044 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-33bda8\"/\"openshift-service-ca.crt\"" Aug 04 00:58:35.627114 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.627057 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-33bda8\"/\"kube-root-ca.crt\"" Aug 04 00:58:35.632322 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.632302 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh"] Aug 04 00:58:35.656446 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.656424 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 00:58:35.719517 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.719488 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df193e3-70bb-4627-9c8f-f2efe3928900-tls-certs\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.719619 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.719528 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xgwq\" (UniqueName: \"kubernetes.io/projected/8df193e3-70bb-4627-9c8f-f2efe3928900-kube-api-access-5xgwq\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.719667 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.719613 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-tmp-dir\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.719667 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.719650 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-dshm\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.719743 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.719691 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-home\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.719743 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.719714 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-model-cache\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.820759 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.820724 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-tmp-dir\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.820862 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.820771 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-dshm\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.820862 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.820810 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-home\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.820862 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.820842 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-model-cache\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.820974 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.820919 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df193e3-70bb-4627-9c8f-f2efe3928900-tls-certs\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.820974 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.820961 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xgwq\" (UniqueName: \"kubernetes.io/projected/8df193e3-70bb-4627-9c8f-f2efe3928900-kube-api-access-5xgwq\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.821664 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.821624 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-tmp-dir\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.821664 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.821624 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-model-cache\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.821842 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.821697 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-home\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.826433 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.823880 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-dshm\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.826433 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.824338 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df193e3-70bb-4627-9c8f-f2efe3928900-tls-certs\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.831336 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.831311 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xgwq\" (UniqueName: \"kubernetes.io/projected/8df193e3-70bb-4627-9c8f-f2efe3928900-kube-api-access-5xgwq\") pod \"rate-test-model-33bda8-kserve-786d4f99f-kvnxh\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:35.934167 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:35.934116 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:36.062508 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:36.062486 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh"] Aug 04 00:58:36.064870 ip-10-0-138-102 kubenswrapper[2570]: W0804 00:58:36.064844 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8df193e3_70bb_4627_9c8f_f2efe3928900.slice/crio-34d1dbefde0f76564f113a155a3b2abcd57878f688a3c4d80665457faafb4d07 WatchSource:0}: Error finding container 34d1dbefde0f76564f113a155a3b2abcd57878f688a3c4d80665457faafb4d07: Status 404 returned error can't find the container with id 34d1dbefde0f76564f113a155a3b2abcd57878f688a3c4d80665457faafb4d07 Aug 04 00:58:36.942687 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:36.942651 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" event={"ID":"8df193e3-70bb-4627-9c8f-f2efe3928900","Type":"ContainerStarted","Data":"0f93c18ab679398a5cc3493c3601bf52058da6ae287506fbfff8d80adbbb9f17"} Aug 04 00:58:36.943096 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:36.942696 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" event={"ID":"8df193e3-70bb-4627-9c8f-f2efe3928900","Type":"ContainerStarted","Data":"34d1dbefde0f76564f113a155a3b2abcd57878f688a3c4d80665457faafb4d07"} Aug 04 00:58:36.962377 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:36.962333 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" podStartSLOduration=1.9623192170000001 podStartE2EDuration="1.962319217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:58:36.96099841 +0000 UTC m=+3363.443664739" watchObservedRunningTime="2026-08-04 00:58:36.962319217 +0000 UTC m=+3363.444985525" Aug 04 00:58:38.230392 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:38.230344 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:58:45.935258 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:45.935201 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:45.935698 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:45.935273 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:45.948112 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:45.948082 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:58:45.996085 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:58:45.996062 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 00:59:15.819940 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:59:15.819904 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:59:23.924031 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:59:23.923994 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 00:59:55.520299 ip-10-0-138-102 kubenswrapper[2570]: I0804 00:59:55.520265 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 01:00:00.869962 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:00.869927 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:01.817111 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:01.817081 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh"] Aug 04 01:00:01.879414 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:01.879385 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:02.751651 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:02.751617 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7541-140011695391152"] Aug 04 01:00:02.755189 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:02.755170 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695391152" Aug 04 01:00:02.757746 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:02.757538 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ddmsh\"" Aug 04 01:00:02.759152 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:02.759125 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7541-140011695391152"] Aug 04 01:00:02.831149 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:02.831125 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qr49\" (UniqueName: \"kubernetes.io/projected/eb982ea4-067c-4a54-b1e3-35fce7505cbc-kube-api-access-4qr49\") pod \"test-curl-7541-140011695391152\" (UID: \"eb982ea4-067c-4a54-b1e3-35fce7505cbc\") " pod="opendatahub/test-curl-7541-140011695391152" Aug 04 01:00:02.932343 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:02.932317 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qr49\" (UniqueName: \"kubernetes.io/projected/eb982ea4-067c-4a54-b1e3-35fce7505cbc-kube-api-access-4qr49\") pod \"test-curl-7541-140011695391152\" (UID: \"eb982ea4-067c-4a54-b1e3-35fce7505cbc\") " pod="opendatahub/test-curl-7541-140011695391152" Aug 04 01:00:02.940627 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:02.940601 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qr49\" (UniqueName: \"kubernetes.io/projected/eb982ea4-067c-4a54-b1e3-35fce7505cbc-kube-api-access-4qr49\") pod \"test-curl-7541-140011695391152\" (UID: \"eb982ea4-067c-4a54-b1e3-35fce7505cbc\") " pod="opendatahub/test-curl-7541-140011695391152" Aug 04 01:00:03.065697 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:03.065648 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695391152" Aug 04 01:00:03.196159 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:03.196136 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7541-140011695391152"] Aug 04 01:00:03.198333 ip-10-0-138-102 kubenswrapper[2570]: W0804 01:00:03.198307 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb982ea4_067c_4a54_b1e3_35fce7505cbc.slice/crio-1d483785c1085afa6f980cd3d7ea77d3837733156000c0d62d5d3681390f5d8d WatchSource:0}: Error finding container 1d483785c1085afa6f980cd3d7ea77d3837733156000c0d62d5d3681390f5d8d: Status 404 returned error can't find the container with id 1d483785c1085afa6f980cd3d7ea77d3837733156000c0d62d5d3681390f5d8d Aug 04 01:00:03.199774 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:03.199757 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 01:00:03.352077 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:03.352014 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7541-140011695391152" event={"ID":"eb982ea4-067c-4a54-b1e3-35fce7505cbc","Type":"ContainerStarted","Data":"1d483785c1085afa6f980cd3d7ea77d3837733156000c0d62d5d3681390f5d8d"} Aug 04 01:00:03.729459 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:03.729387 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 01:00:05.363767 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:05.363728 2570 generic.go:356] "Generic (PLEG): container finished" podID="eb982ea4-067c-4a54-b1e3-35fce7505cbc" containerID="0f6284882b998cd248f38b9793428ca14fabe71a28e93fa514737fd8508c714c" exitCode=0 Aug 04 01:00:05.364141 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:05.363802 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7541-140011695391152" event={"ID":"eb982ea4-067c-4a54-b1e3-35fce7505cbc","Type":"ContainerDied","Data":"0f6284882b998cd248f38b9793428ca14fabe71a28e93fa514737fd8508c714c"} Aug 04 01:00:06.496624 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:06.496596 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695391152" Aug 04 01:00:06.514319 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:06.514295 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7541-140011695391152_eb982ea4-067c-4a54-b1e3-35fce7505cbc/test-curl-7541-140011695391152/0.log" Aug 04 01:00:06.535021 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:06.534994 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7541-140011695391152"] Aug 04 01:00:06.536494 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:06.536476 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7541-140011695391152"] Aug 04 01:00:06.560975 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:06.560953 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qr49\" (UniqueName: \"kubernetes.io/projected/eb982ea4-067c-4a54-b1e3-35fce7505cbc-kube-api-access-4qr49\") pod \"eb982ea4-067c-4a54-b1e3-35fce7505cbc\" (UID: \"eb982ea4-067c-4a54-b1e3-35fce7505cbc\") " Aug 04 01:00:06.562962 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:06.562943 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb982ea4-067c-4a54-b1e3-35fce7505cbc-kube-api-access-4qr49" (OuterVolumeSpecName: "kube-api-access-4qr49") pod "eb982ea4-067c-4a54-b1e3-35fce7505cbc" (UID: "eb982ea4-067c-4a54-b1e3-35fce7505cbc"). InnerVolumeSpecName "kube-api-access-4qr49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:00:06.658044 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:06.657985 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7541-140011695393968"] Aug 04 01:00:06.658354 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:06.658340 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb982ea4-067c-4a54-b1e3-35fce7505cbc" containerName="test-curl-7541-140011695391152" Aug 04 01:00:06.658403 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:06.658355 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb982ea4-067c-4a54-b1e3-35fce7505cbc" containerName="test-curl-7541-140011695391152" Aug 04 01:00:06.658439 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:06.658427 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb982ea4-067c-4a54-b1e3-35fce7505cbc" containerName="test-curl-7541-140011695391152" Aug 04 01:00:06.661484 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:06.661469 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695393968" Aug 04 01:00:06.661785 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:06.661763 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qr49\" (UniqueName: \"kubernetes.io/projected/eb982ea4-067c-4a54-b1e3-35fce7505cbc-kube-api-access-4qr49\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:00:06.667537 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:06.667514 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7541-140011695393968"] Aug 04 01:00:06.762384 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:06.762351 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr5ct\" (UniqueName: \"kubernetes.io/projected/d09099c3-9d08-4ba8-800d-126c79fa2d4b-kube-api-access-kr5ct\") pod \"test-curl-7541-140011695393968\" (UID: \"d09099c3-9d08-4ba8-800d-126c79fa2d4b\") " pod="opendatahub/test-curl-7541-140011695393968" Aug 04 01:00:06.863567 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:06.863540 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr5ct\" (UniqueName: \"kubernetes.io/projected/d09099c3-9d08-4ba8-800d-126c79fa2d4b-kube-api-access-kr5ct\") pod \"test-curl-7541-140011695393968\" (UID: \"d09099c3-9d08-4ba8-800d-126c79fa2d4b\") " pod="opendatahub/test-curl-7541-140011695393968" Aug 04 01:00:06.871688 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:06.871661 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr5ct\" (UniqueName: \"kubernetes.io/projected/d09099c3-9d08-4ba8-800d-126c79fa2d4b-kube-api-access-kr5ct\") pod \"test-curl-7541-140011695393968\" (UID: \"d09099c3-9d08-4ba8-800d-126c79fa2d4b\") " pod="opendatahub/test-curl-7541-140011695393968" Aug 04 01:00:06.970692 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:06.970637 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695393968" Aug 04 01:00:07.103501 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:07.103410 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7541-140011695393968"] Aug 04 01:00:07.106008 ip-10-0-138-102 kubenswrapper[2570]: W0804 01:00:07.105984 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd09099c3_9d08_4ba8_800d_126c79fa2d4b.slice/crio-9f4a6908dc02fbdcd18de8fda74f2238fb4f1da287ccde854ff3c2f3068d0d7b WatchSource:0}: Error finding container 9f4a6908dc02fbdcd18de8fda74f2238fb4f1da287ccde854ff3c2f3068d0d7b: Status 404 returned error can't find the container with id 9f4a6908dc02fbdcd18de8fda74f2238fb4f1da287ccde854ff3c2f3068d0d7b Aug 04 01:00:07.374532 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:07.374501 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7541-140011695393968" event={"ID":"d09099c3-9d08-4ba8-800d-126c79fa2d4b","Type":"ContainerStarted","Data":"9f4a6908dc02fbdcd18de8fda74f2238fb4f1da287ccde854ff3c2f3068d0d7b"} Aug 04 01:00:07.375689 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:07.375672 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695391152" Aug 04 01:00:07.375787 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:07.375712 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d483785c1085afa6f980cd3d7ea77d3837733156000c0d62d5d3681390f5d8d" Aug 04 01:00:07.385983 ip-10-0-138-102 kubenswrapper[2570]: E0804 01:00:07.385958 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7541-140011695391152\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="eb982ea4-067c-4a54-b1e3-35fce7505cbc" pod="opendatahub/test-curl-7541-140011695391152" Aug 04 01:00:08.034616 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:08.034582 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb982ea4-067c-4a54-b1e3-35fce7505cbc" path="/var/lib/kubelet/pods/eb982ea4-067c-4a54-b1e3-35fce7505cbc/volumes" Aug 04 01:00:08.381894 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:08.381814 2570 generic.go:356] "Generic (PLEG): container finished" podID="d09099c3-9d08-4ba8-800d-126c79fa2d4b" containerID="793175b4d8dd15600c2d9853aa0e58b7c7bd4715da3623184f041edd253f1a4e" exitCode=0 Aug 04 01:00:08.382050 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:08.381895 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7541-140011695393968" event={"ID":"d09099c3-9d08-4ba8-800d-126c79fa2d4b","Type":"ContainerDied","Data":"793175b4d8dd15600c2d9853aa0e58b7c7bd4715da3623184f041edd253f1a4e"} Aug 04 01:00:09.518217 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:09.518191 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695393968" Aug 04 01:00:09.536355 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:09.536331 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7541-140011695393968_d09099c3-9d08-4ba8-800d-126c79fa2d4b/test-curl-7541-140011695393968/0.log" Aug 04 01:00:09.555071 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:09.555047 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7541-140011695393968"] Aug 04 01:00:09.557020 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:09.557000 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7541-140011695393968"] Aug 04 01:00:09.587405 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:09.587383 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr5ct\" (UniqueName: \"kubernetes.io/projected/d09099c3-9d08-4ba8-800d-126c79fa2d4b-kube-api-access-kr5ct\") pod \"d09099c3-9d08-4ba8-800d-126c79fa2d4b\" (UID: \"d09099c3-9d08-4ba8-800d-126c79fa2d4b\") " Aug 04 01:00:09.589357 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:09.589329 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09099c3-9d08-4ba8-800d-126c79fa2d4b-kube-api-access-kr5ct" (OuterVolumeSpecName: "kube-api-access-kr5ct") pod "d09099c3-9d08-4ba8-800d-126c79fa2d4b" (UID: "d09099c3-9d08-4ba8-800d-126c79fa2d4b"). InnerVolumeSpecName "kube-api-access-kr5ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:00:09.688247 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:09.688180 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr5ct\" (UniqueName: \"kubernetes.io/projected/d09099c3-9d08-4ba8-800d-126c79fa2d4b-kube-api-access-kr5ct\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:00:09.721586 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:09.721564 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7541-140011695357216"] Aug 04 01:00:09.721931 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:09.721919 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d09099c3-9d08-4ba8-800d-126c79fa2d4b" containerName="test-curl-7541-140011695393968" Aug 04 01:00:09.721975 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:09.721933 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d09099c3-9d08-4ba8-800d-126c79fa2d4b" containerName="test-curl-7541-140011695393968" Aug 04 01:00:09.722011 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:09.721986 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d09099c3-9d08-4ba8-800d-126c79fa2d4b" containerName="test-curl-7541-140011695393968" Aug 04 01:00:09.724926 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:09.724912 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695357216" Aug 04 01:00:09.731141 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:09.731118 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7541-140011695357216"] Aug 04 01:00:09.788762 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:09.788738 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l529s\" (UniqueName: \"kubernetes.io/projected/469a1080-b299-4994-8eab-1e9923290dee-kube-api-access-l529s\") pod \"test-curl-7541-140011695357216\" (UID: \"469a1080-b299-4994-8eab-1e9923290dee\") " pod="opendatahub/test-curl-7541-140011695357216" Aug 04 01:00:09.889874 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:09.889853 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l529s\" (UniqueName: \"kubernetes.io/projected/469a1080-b299-4994-8eab-1e9923290dee-kube-api-access-l529s\") pod \"test-curl-7541-140011695357216\" (UID: \"469a1080-b299-4994-8eab-1e9923290dee\") " pod="opendatahub/test-curl-7541-140011695357216" Aug 04 01:00:09.897732 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:09.897712 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l529s\" (UniqueName: \"kubernetes.io/projected/469a1080-b299-4994-8eab-1e9923290dee-kube-api-access-l529s\") pod \"test-curl-7541-140011695357216\" (UID: \"469a1080-b299-4994-8eab-1e9923290dee\") " pod="opendatahub/test-curl-7541-140011695357216" Aug 04 01:00:10.034130 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:10.034105 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d09099c3-9d08-4ba8-800d-126c79fa2d4b" path="/var/lib/kubelet/pods/d09099c3-9d08-4ba8-800d-126c79fa2d4b/volumes" Aug 04 01:00:10.034575 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:10.034552 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695357216" Aug 04 01:00:10.161532 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:10.161461 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7541-140011695357216"] Aug 04 01:00:10.163694 ip-10-0-138-102 kubenswrapper[2570]: W0804 01:00:10.163662 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod469a1080_b299_4994_8eab_1e9923290dee.slice/crio-a5f9b51d4d72af2269642590943e7a70c74f43f8291fe57b1f989f84531b5b20 WatchSource:0}: Error finding container a5f9b51d4d72af2269642590943e7a70c74f43f8291fe57b1f989f84531b5b20: Status 404 returned error can't find the container with id a5f9b51d4d72af2269642590943e7a70c74f43f8291fe57b1f989f84531b5b20 Aug 04 01:00:10.393070 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:10.392995 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7541-140011695357216" event={"ID":"469a1080-b299-4994-8eab-1e9923290dee","Type":"ContainerStarted","Data":"a5f9b51d4d72af2269642590943e7a70c74f43f8291fe57b1f989f84531b5b20"} Aug 04 01:00:10.394233 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:10.394211 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695393968" Aug 04 01:00:10.394334 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:10.394215 2570 scope.go:117] "RemoveContainer" containerID="793175b4d8dd15600c2d9853aa0e58b7c7bd4715da3623184f041edd253f1a4e" Aug 04 01:00:11.406957 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:11.406915 2570 generic.go:356] "Generic (PLEG): container finished" podID="469a1080-b299-4994-8eab-1e9923290dee" containerID="b11f7673626e4f955c0a7cf8e69e88c6ce2ccafd83d2d515c679234b830598ab" exitCode=0 Aug 04 01:00:11.407382 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:11.406994 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7541-140011695357216" event={"ID":"469a1080-b299-4994-8eab-1e9923290dee","Type":"ContainerDied","Data":"b11f7673626e4f955c0a7cf8e69e88c6ce2ccafd83d2d515c679234b830598ab"} Aug 04 01:00:12.543419 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:12.543397 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695357216" Aug 04 01:00:12.561998 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:12.561966 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7541-140011695357216_469a1080-b299-4994-8eab-1e9923290dee/test-curl-7541-140011695357216/0.log" Aug 04 01:00:12.581779 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:12.581755 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7541-140011695357216"] Aug 04 01:00:12.585001 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:12.584981 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7541-140011695357216"] Aug 04 01:00:12.613605 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:12.613584 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l529s\" (UniqueName: \"kubernetes.io/projected/469a1080-b299-4994-8eab-1e9923290dee-kube-api-access-l529s\") pod \"469a1080-b299-4994-8eab-1e9923290dee\" (UID: \"469a1080-b299-4994-8eab-1e9923290dee\") " Aug 04 01:00:12.615547 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:12.615528 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469a1080-b299-4994-8eab-1e9923290dee-kube-api-access-l529s" (OuterVolumeSpecName: "kube-api-access-l529s") pod "469a1080-b299-4994-8eab-1e9923290dee" (UID: "469a1080-b299-4994-8eab-1e9923290dee"). InnerVolumeSpecName "kube-api-access-l529s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:00:12.699306 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:12.699213 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7541-140011695357216"] Aug 04 01:00:12.699692 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:12.699676 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469a1080-b299-4994-8eab-1e9923290dee" containerName="test-curl-7541-140011695357216" Aug 04 01:00:12.699774 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:12.699695 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="469a1080-b299-4994-8eab-1e9923290dee" containerName="test-curl-7541-140011695357216" Aug 04 01:00:12.699828 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:12.699796 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="469a1080-b299-4994-8eab-1e9923290dee" containerName="test-curl-7541-140011695357216" Aug 04 01:00:12.702776 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:12.702758 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695357216" Aug 04 01:00:12.708506 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:12.708486 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7541-140011695357216"] Aug 04 01:00:12.715177 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:12.715150 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l529s\" (UniqueName: \"kubernetes.io/projected/469a1080-b299-4994-8eab-1e9923290dee-kube-api-access-l529s\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:00:12.816700 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:12.816675 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr9qj\" (UniqueName: \"kubernetes.io/projected/7d943aa1-44ef-4f45-97ff-362e05dc05a5-kube-api-access-vr9qj\") pod \"test-curl-7541-140011695357216\" (UID: \"7d943aa1-44ef-4f45-97ff-362e05dc05a5\") " pod="opendatahub/test-curl-7541-140011695357216" Aug 04 01:00:12.917285 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:12.917251 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr9qj\" (UniqueName: \"kubernetes.io/projected/7d943aa1-44ef-4f45-97ff-362e05dc05a5-kube-api-access-vr9qj\") pod \"test-curl-7541-140011695357216\" (UID: \"7d943aa1-44ef-4f45-97ff-362e05dc05a5\") " pod="opendatahub/test-curl-7541-140011695357216" Aug 04 01:00:12.924998 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:12.924964 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr9qj\" (UniqueName: \"kubernetes.io/projected/7d943aa1-44ef-4f45-97ff-362e05dc05a5-kube-api-access-vr9qj\") pod \"test-curl-7541-140011695357216\" (UID: \"7d943aa1-44ef-4f45-97ff-362e05dc05a5\") " pod="opendatahub/test-curl-7541-140011695357216" Aug 04 01:00:13.013884 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:13.013864 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695357216" Aug 04 01:00:13.346283 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:13.346254 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7541-140011695357216"] Aug 04 01:00:13.348253 ip-10-0-138-102 kubenswrapper[2570]: W0804 01:00:13.348211 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d943aa1_44ef_4f45_97ff_362e05dc05a5.slice/crio-22fc410eee47a97fdadf899f46f3e54ee633114db6c7d7099205504e1398e965 WatchSource:0}: Error finding container 22fc410eee47a97fdadf899f46f3e54ee633114db6c7d7099205504e1398e965: Status 404 returned error can't find the container with id 22fc410eee47a97fdadf899f46f3e54ee633114db6c7d7099205504e1398e965 Aug 04 01:00:13.418357 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:13.418330 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695357216" Aug 04 01:00:13.418448 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:13.418356 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5f9b51d4d72af2269642590943e7a70c74f43f8291fe57b1f989f84531b5b20" Aug 04 01:00:13.419521 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:13.419500 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7541-140011695357216" event={"ID":"7d943aa1-44ef-4f45-97ff-362e05dc05a5","Type":"ContainerStarted","Data":"22fc410eee47a97fdadf899f46f3e54ee633114db6c7d7099205504e1398e965"} Aug 04 01:00:13.430118 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:13.430086 2570 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7541-140011695357216" oldPodUID="469a1080-b299-4994-8eab-1e9923290dee" podUID="7d943aa1-44ef-4f45-97ff-362e05dc05a5" Aug 04 01:00:14.034114 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:14.034086 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="469a1080-b299-4994-8eab-1e9923290dee" path="/var/lib/kubelet/pods/469a1080-b299-4994-8eab-1e9923290dee/volumes" Aug 04 01:00:14.429942 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:14.429853 2570 generic.go:356] "Generic (PLEG): container finished" podID="7d943aa1-44ef-4f45-97ff-362e05dc05a5" containerID="b4730d69ef5038cea22244c92261c34c4ab6faaff8dca85d9b848a06a33ea29b" exitCode=0 Aug 04 01:00:14.430129 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:14.429939 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7541-140011695357216" event={"ID":"7d943aa1-44ef-4f45-97ff-362e05dc05a5","Type":"ContainerDied","Data":"b4730d69ef5038cea22244c92261c34c4ab6faaff8dca85d9b848a06a33ea29b"} Aug 04 01:00:15.562088 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:15.562066 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695357216" Aug 04 01:00:15.581232 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:15.581200 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7541-140011695357216_7d943aa1-44ef-4f45-97ff-362e05dc05a5/test-curl-7541-140011695357216/0.log" Aug 04 01:00:15.605151 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:15.605127 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7541-140011695357216"] Aug 04 01:00:15.608846 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:15.608826 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7541-140011695357216"] Aug 04 01:00:15.638617 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:15.638597 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr9qj\" (UniqueName: \"kubernetes.io/projected/7d943aa1-44ef-4f45-97ff-362e05dc05a5-kube-api-access-vr9qj\") pod \"7d943aa1-44ef-4f45-97ff-362e05dc05a5\" (UID: \"7d943aa1-44ef-4f45-97ff-362e05dc05a5\") " Aug 04 01:00:15.640570 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:15.640546 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d943aa1-44ef-4f45-97ff-362e05dc05a5-kube-api-access-vr9qj" (OuterVolumeSpecName: "kube-api-access-vr9qj") pod "7d943aa1-44ef-4f45-97ff-362e05dc05a5" (UID: "7d943aa1-44ef-4f45-97ff-362e05dc05a5"). InnerVolumeSpecName "kube-api-access-vr9qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:00:15.730133 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:15.730072 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7541-140011695350096"] Aug 04 01:00:15.730521 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:15.730504 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d943aa1-44ef-4f45-97ff-362e05dc05a5" containerName="test-curl-7541-140011695357216" Aug 04 01:00:15.730594 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:15.730523 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d943aa1-44ef-4f45-97ff-362e05dc05a5" containerName="test-curl-7541-140011695357216" Aug 04 01:00:15.730663 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:15.730651 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d943aa1-44ef-4f45-97ff-362e05dc05a5" containerName="test-curl-7541-140011695357216" Aug 04 01:00:15.733466 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:15.733449 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695350096" Aug 04 01:00:15.737406 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:15.737385 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7541-140011695350096"] Aug 04 01:00:15.739484 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:15.739460 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr9qj\" (UniqueName: \"kubernetes.io/projected/7d943aa1-44ef-4f45-97ff-362e05dc05a5-kube-api-access-vr9qj\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:00:15.846542 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:15.840925 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9rsk\" (UniqueName: \"kubernetes.io/projected/3c3fb56d-45af-4a4b-8c3c-7805a80575ee-kube-api-access-v9rsk\") pod \"test-curl-7541-140011695350096\" (UID: \"3c3fb56d-45af-4a4b-8c3c-7805a80575ee\") " pod="opendatahub/test-curl-7541-140011695350096" Aug 04 01:00:15.942053 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:15.942020 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9rsk\" (UniqueName: \"kubernetes.io/projected/3c3fb56d-45af-4a4b-8c3c-7805a80575ee-kube-api-access-v9rsk\") pod \"test-curl-7541-140011695350096\" (UID: \"3c3fb56d-45af-4a4b-8c3c-7805a80575ee\") " pod="opendatahub/test-curl-7541-140011695350096" Aug 04 01:00:15.950419 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:15.950393 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9rsk\" (UniqueName: \"kubernetes.io/projected/3c3fb56d-45af-4a4b-8c3c-7805a80575ee-kube-api-access-v9rsk\") pod \"test-curl-7541-140011695350096\" (UID: \"3c3fb56d-45af-4a4b-8c3c-7805a80575ee\") " pod="opendatahub/test-curl-7541-140011695350096" Aug 04 01:00:16.033725 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:16.033690 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d943aa1-44ef-4f45-97ff-362e05dc05a5" path="/var/lib/kubelet/pods/7d943aa1-44ef-4f45-97ff-362e05dc05a5/volumes" Aug 04 01:00:16.043923 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:16.043902 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695350096" Aug 04 01:00:16.168604 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:16.168583 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7541-140011695350096"] Aug 04 01:00:16.170575 ip-10-0-138-102 kubenswrapper[2570]: W0804 01:00:16.170550 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c3fb56d_45af_4a4b_8c3c_7805a80575ee.slice/crio-f879c5a5340a28a6dba31b9eb63af6d197037ca3c1786156eb378c0d2f861d9b WatchSource:0}: Error finding container f879c5a5340a28a6dba31b9eb63af6d197037ca3c1786156eb378c0d2f861d9b: Status 404 returned error can't find the container with id f879c5a5340a28a6dba31b9eb63af6d197037ca3c1786156eb378c0d2f861d9b Aug 04 01:00:16.441642 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:16.441610 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7541-140011695350096" event={"ID":"3c3fb56d-45af-4a4b-8c3c-7805a80575ee","Type":"ContainerStarted","Data":"f879c5a5340a28a6dba31b9eb63af6d197037ca3c1786156eb378c0d2f861d9b"} Aug 04 01:00:16.442835 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:16.442816 2570 scope.go:117] "RemoveContainer" containerID="b4730d69ef5038cea22244c92261c34c4ab6faaff8dca85d9b848a06a33ea29b" Aug 04 01:00:16.442835 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:16.442832 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695357216" Aug 04 01:00:16.835643 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:16.835599 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" podUID="8df193e3-70bb-4627-9c8f-f2efe3928900" containerName="main" containerID="cri-o://0f93c18ab679398a5cc3493c3601bf52058da6ae287506fbfff8d80adbbb9f17" gracePeriod=45 Aug 04 01:00:17.091925 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.091867 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 01:00:17.151814 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.151784 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-tmp-dir\") pod \"8df193e3-70bb-4627-9c8f-f2efe3928900\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " Aug 04 01:00:17.151814 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.151813 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-model-cache\") pod \"8df193e3-70bb-4627-9c8f-f2efe3928900\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " Aug 04 01:00:17.152039 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.151837 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xgwq\" (UniqueName: \"kubernetes.io/projected/8df193e3-70bb-4627-9c8f-f2efe3928900-kube-api-access-5xgwq\") pod \"8df193e3-70bb-4627-9c8f-f2efe3928900\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " Aug 04 01:00:17.152039 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.151853 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-dshm\") pod \"8df193e3-70bb-4627-9c8f-f2efe3928900\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " Aug 04 01:00:17.152039 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.151871 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df193e3-70bb-4627-9c8f-f2efe3928900-tls-certs\") pod \"8df193e3-70bb-4627-9c8f-f2efe3928900\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " Aug 04 01:00:17.152039 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.151898 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-home\") pod \"8df193e3-70bb-4627-9c8f-f2efe3928900\" (UID: \"8df193e3-70bb-4627-9c8f-f2efe3928900\") " Aug 04 01:00:17.152283 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.152078 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8df193e3-70bb-4627-9c8f-f2efe3928900" (UID: "8df193e3-70bb-4627-9c8f-f2efe3928900"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:00:17.152283 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.152088 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-model-cache" (OuterVolumeSpecName: "model-cache") pod "8df193e3-70bb-4627-9c8f-f2efe3928900" (UID: "8df193e3-70bb-4627-9c8f-f2efe3928900"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:00:17.152283 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.152170 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-home" (OuterVolumeSpecName: "home") pod "8df193e3-70bb-4627-9c8f-f2efe3928900" (UID: "8df193e3-70bb-4627-9c8f-f2efe3928900"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:00:17.152283 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.152212 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:00:17.152283 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.152261 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:00:17.153823 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.153798 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df193e3-70bb-4627-9c8f-f2efe3928900-kube-api-access-5xgwq" (OuterVolumeSpecName: "kube-api-access-5xgwq") pod "8df193e3-70bb-4627-9c8f-f2efe3928900" (UID: "8df193e3-70bb-4627-9c8f-f2efe3928900"). InnerVolumeSpecName "kube-api-access-5xgwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:00:17.153939 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.153920 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-dshm" (OuterVolumeSpecName: "dshm") pod "8df193e3-70bb-4627-9c8f-f2efe3928900" (UID: "8df193e3-70bb-4627-9c8f-f2efe3928900"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:00:17.153996 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.153914 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df193e3-70bb-4627-9c8f-f2efe3928900-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8df193e3-70bb-4627-9c8f-f2efe3928900" (UID: "8df193e3-70bb-4627-9c8f-f2efe3928900"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 01:00:17.252900 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.252874 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xgwq\" (UniqueName: \"kubernetes.io/projected/8df193e3-70bb-4627-9c8f-f2efe3928900-kube-api-access-5xgwq\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:00:17.252900 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.252897 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:00:17.253020 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.252907 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df193e3-70bb-4627-9c8f-f2efe3928900-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:00:17.253020 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.252916 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df193e3-70bb-4627-9c8f-f2efe3928900-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:00:17.450381 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.450306 2570 generic.go:356] "Generic (PLEG): container finished" podID="8df193e3-70bb-4627-9c8f-f2efe3928900" containerID="0f93c18ab679398a5cc3493c3601bf52058da6ae287506fbfff8d80adbbb9f17" exitCode=0 Aug 04 01:00:17.450381 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.450346 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" Aug 04 01:00:17.450571 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.450390 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" event={"ID":"8df193e3-70bb-4627-9c8f-f2efe3928900","Type":"ContainerDied","Data":"0f93c18ab679398a5cc3493c3601bf52058da6ae287506fbfff8d80adbbb9f17"} Aug 04 01:00:17.450571 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.450438 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh" event={"ID":"8df193e3-70bb-4627-9c8f-f2efe3928900","Type":"ContainerDied","Data":"34d1dbefde0f76564f113a155a3b2abcd57878f688a3c4d80665457faafb4d07"} Aug 04 01:00:17.450571 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.450460 2570 scope.go:117] "RemoveContainer" containerID="0f93c18ab679398a5cc3493c3601bf52058da6ae287506fbfff8d80adbbb9f17" Aug 04 01:00:17.452009 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.451987 2570 generic.go:356] "Generic (PLEG): container finished" podID="3c3fb56d-45af-4a4b-8c3c-7805a80575ee" containerID="0d59d5b4185c8f53c18b593b00189758a1b396ad39715d2ea035396cedd9c5db" exitCode=0 Aug 04 01:00:17.452113 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.452023 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7541-140011695350096" event={"ID":"3c3fb56d-45af-4a4b-8c3c-7805a80575ee","Type":"ContainerDied","Data":"0d59d5b4185c8f53c18b593b00189758a1b396ad39715d2ea035396cedd9c5db"} Aug 04 01:00:17.460550 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.460532 2570 scope.go:117] "RemoveContainer" containerID="0f93c18ab679398a5cc3493c3601bf52058da6ae287506fbfff8d80adbbb9f17" Aug 04 01:00:17.460808 ip-10-0-138-102 kubenswrapper[2570]: E0804 01:00:17.460781 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f93c18ab679398a5cc3493c3601bf52058da6ae287506fbfff8d80adbbb9f17\": container with ID starting with 0f93c18ab679398a5cc3493c3601bf52058da6ae287506fbfff8d80adbbb9f17 not found: ID does not exist" containerID="0f93c18ab679398a5cc3493c3601bf52058da6ae287506fbfff8d80adbbb9f17" Aug 04 01:00:17.460871 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.460820 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f93c18ab679398a5cc3493c3601bf52058da6ae287506fbfff8d80adbbb9f17"} err="failed to get container status \"0f93c18ab679398a5cc3493c3601bf52058da6ae287506fbfff8d80adbbb9f17\": rpc error: code = NotFound desc = could not find container \"0f93c18ab679398a5cc3493c3601bf52058da6ae287506fbfff8d80adbbb9f17\": container with ID starting with 0f93c18ab679398a5cc3493c3601bf52058da6ae287506fbfff8d80adbbb9f17 not found: ID does not exist" Aug 04 01:00:17.477013 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.476987 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh"] Aug 04 01:00:17.480847 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:17.480828 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-33bda8/rate-test-model-33bda8-kserve-786d4f99f-kvnxh"] Aug 04 01:00:18.034052 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:18.034016 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8df193e3-70bb-4627-9c8f-f2efe3928900" path="/var/lib/kubelet/pods/8df193e3-70bb-4627-9c8f-f2efe3928900/volumes" Aug 04 01:00:18.591744 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:18.591725 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695350096" Aug 04 01:00:18.610394 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:18.610367 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7541-140011695350096_3c3fb56d-45af-4a4b-8c3c-7805a80575ee/test-curl-7541-140011695350096/0.log" Aug 04 01:00:18.639045 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:18.639020 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7541-140011695350096"] Aug 04 01:00:18.644693 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:18.644674 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7541-140011695350096"] Aug 04 01:00:18.664744 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:18.664722 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9rsk\" (UniqueName: \"kubernetes.io/projected/3c3fb56d-45af-4a4b-8c3c-7805a80575ee-kube-api-access-v9rsk\") pod \"3c3fb56d-45af-4a4b-8c3c-7805a80575ee\" (UID: \"3c3fb56d-45af-4a4b-8c3c-7805a80575ee\") " Aug 04 01:00:18.666572 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:18.666551 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3fb56d-45af-4a4b-8c3c-7805a80575ee-kube-api-access-v9rsk" (OuterVolumeSpecName: "kube-api-access-v9rsk") pod "3c3fb56d-45af-4a4b-8c3c-7805a80575ee" (UID: "3c3fb56d-45af-4a4b-8c3c-7805a80575ee"). InnerVolumeSpecName "kube-api-access-v9rsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:00:18.766091 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:18.766070 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9rsk\" (UniqueName: \"kubernetes.io/projected/3c3fb56d-45af-4a4b-8c3c-7805a80575ee-kube-api-access-v9rsk\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:00:19.465377 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:19.465347 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f879c5a5340a28a6dba31b9eb63af6d197037ca3c1786156eb378c0d2f861d9b" Aug 04 01:00:19.465832 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:19.465391 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-140011695350096" Aug 04 01:00:19.475609 ip-10-0-138-102 kubenswrapper[2570]: E0804 01:00:19.475568 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7541-140011695350096\" is forbidden: User \"system:node:ip-10-0-138-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-102.ec2.internal' and this object" podUID="3c3fb56d-45af-4a4b-8c3c-7805a80575ee" pod="opendatahub/test-curl-7541-140011695350096" Aug 04 01:00:20.033824 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:20.033793 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c3fb56d-45af-4a4b-8c3c-7805a80575ee" path="/var/lib/kubelet/pods/3c3fb56d-45af-4a4b-8c3c-7805a80575ee/volumes" Aug 04 01:00:21.900207 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:21.900154 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:22.197621 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:22.197426 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:22.719803 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:22.719770 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:25.117392 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:25.117356 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:25.218477 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:25.218444 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:28.113868 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.113823 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx"] Aug 04 01:00:28.114826 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.114794 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3fb56d-45af-4a4b-8c3c-7805a80575ee" containerName="test-curl-7541-140011695350096" Aug 04 01:00:28.114826 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.114821 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3fb56d-45af-4a4b-8c3c-7805a80575ee" containerName="test-curl-7541-140011695350096" Aug 04 01:00:28.115008 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.114844 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8df193e3-70bb-4627-9c8f-f2efe3928900" containerName="main" Aug 04 01:00:28.115008 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.114851 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8df193e3-70bb-4627-9c8f-f2efe3928900" containerName="main" Aug 04 01:00:28.115008 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.114915 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3fb56d-45af-4a4b-8c3c-7805a80575ee" containerName="test-curl-7541-140011695350096" Aug 04 01:00:28.115008 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.114925 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8df193e3-70bb-4627-9c8f-f2efe3928900" containerName="main" Aug 04 01:00:28.120266 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.120217 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" Aug 04 01:00:28.123950 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.123779 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0825a6-serving-cert\"" Aug 04 01:00:28.123950 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.123795 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 01:00:28.123950 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.123821 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pvrls\"" Aug 04 01:00:28.124189 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.124085 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 01:00:28.126874 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.126847 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx"] Aug 04 01:00:28.153296 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.153273 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:28.244028 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.244000 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1520aab-8f0c-45e8-b598-aa68d9370b3e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx\" (UID: \"a1520aab-8f0c-45e8-b598-aa68d9370b3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" Aug 04 01:00:28.244109 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.244060 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgptz\" (UniqueName: \"kubernetes.io/projected/a1520aab-8f0c-45e8-b598-aa68d9370b3e-kube-api-access-hgptz\") pod \"maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx\" (UID: \"a1520aab-8f0c-45e8-b598-aa68d9370b3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" Aug 04 01:00:28.244109 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.244100 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1520aab-8f0c-45e8-b598-aa68d9370b3e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx\" (UID: \"a1520aab-8f0c-45e8-b598-aa68d9370b3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" Aug 04 01:00:28.350896 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.350865 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1520aab-8f0c-45e8-b598-aa68d9370b3e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx\" (UID: \"a1520aab-8f0c-45e8-b598-aa68d9370b3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" Aug 04 01:00:28.350997 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.350954 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1520aab-8f0c-45e8-b598-aa68d9370b3e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx\" (UID: \"a1520aab-8f0c-45e8-b598-aa68d9370b3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" Aug 04 01:00:28.351051 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.351002 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgptz\" (UniqueName: \"kubernetes.io/projected/a1520aab-8f0c-45e8-b598-aa68d9370b3e-kube-api-access-hgptz\") pod \"maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx\" (UID: \"a1520aab-8f0c-45e8-b598-aa68d9370b3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" Aug 04 01:00:28.351708 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.351685 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1520aab-8f0c-45e8-b598-aa68d9370b3e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx\" (UID: \"a1520aab-8f0c-45e8-b598-aa68d9370b3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" Aug 04 01:00:28.353467 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.353449 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1520aab-8f0c-45e8-b598-aa68d9370b3e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx\" (UID: \"a1520aab-8f0c-45e8-b598-aa68d9370b3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" Aug 04 01:00:28.371752 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.371699 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgptz\" (UniqueName: \"kubernetes.io/projected/a1520aab-8f0c-45e8-b598-aa68d9370b3e-kube-api-access-hgptz\") pod \"maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx\" (UID: \"a1520aab-8f0c-45e8-b598-aa68d9370b3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" Aug 04 01:00:28.436295 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.436273 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" Aug 04 01:00:28.451380 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.451339 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:28.564203 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.564177 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx"] Aug 04 01:00:28.566971 ip-10-0-138-102 kubenswrapper[2570]: W0804 01:00:28.566949 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1520aab_8f0c_45e8_b598_aa68d9370b3e.slice/crio-6ad968de5e753698a7822316b71cfcd42798e8130614c2edf88e07fedd9f1228 WatchSource:0}: Error finding container 6ad968de5e753698a7822316b71cfcd42798e8130614c2edf88e07fedd9f1228: Status 404 returned error can't find the container with id 6ad968de5e753698a7822316b71cfcd42798e8130614c2edf88e07fedd9f1228 Aug 04 01:00:28.786126 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.786094 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:28.887498 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:28.887468 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:29.519555 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:29.519521 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" event={"ID":"a1520aab-8f0c-45e8-b598-aa68d9370b3e","Type":"ContainerStarted","Data":"6ad968de5e753698a7822316b71cfcd42798e8130614c2edf88e07fedd9f1228"} Aug 04 01:00:30.528259 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:30.528206 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" event={"ID":"a1520aab-8f0c-45e8-b598-aa68d9370b3e","Type":"ContainerStarted","Data":"e9083bad8f5c1794a2c699c59cf4764faf9e934862d4243e8e361e1cbd10829d"} Aug 04 01:00:30.528626 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:30.528319 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" Aug 04 01:00:30.543725 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:30.543686 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" podStartSLOduration=1.137579012 podStartE2EDuration="2.543673161s" totalImagesPullingTime="1.406094149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 01:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:00:30.542532079 +0000 UTC m=+3477.025198410" watchObservedRunningTime="2026-08-04 01:00:30.543673161 +0000 UTC m=+3477.026339497" Aug 04 01:00:31.039783 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:31.039747 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 01:00:36.540689 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:36.540657 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" Aug 04 01:00:39.846303 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:39.846269 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:45.620058 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:45.620010 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:45.922106 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:45.922021 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:46.056918 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:46.056874 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:49.304448 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:49.304409 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:49.402579 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:49.402534 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:51.001594 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.001560 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md"] Aug 04 01:00:51.005639 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.005622 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" Aug 04 01:00:51.007916 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.007889 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-1cbf8c-serving-cert\"" Aug 04 01:00:51.016015 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.015997 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md"] Aug 04 01:00:51.031107 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.031063 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:51.139768 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.139743 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2xf4\" (UniqueName: \"kubernetes.io/projected/e3f445db-61b7-4dcc-967d-4cf4d79ee302-kube-api-access-x2xf4\") pod \"maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md\" (UID: \"e3f445db-61b7-4dcc-967d-4cf4d79ee302\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" Aug 04 01:00:51.139872 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.139791 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3f445db-61b7-4dcc-967d-4cf4d79ee302-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md\" (UID: \"e3f445db-61b7-4dcc-967d-4cf4d79ee302\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" Aug 04 01:00:51.139921 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.139862 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3f445db-61b7-4dcc-967d-4cf4d79ee302-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md\" (UID: \"e3f445db-61b7-4dcc-967d-4cf4d79ee302\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" Aug 04 01:00:51.240507 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.240478 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2xf4\" (UniqueName: \"kubernetes.io/projected/e3f445db-61b7-4dcc-967d-4cf4d79ee302-kube-api-access-x2xf4\") pod \"maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md\" (UID: \"e3f445db-61b7-4dcc-967d-4cf4d79ee302\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" Aug 04 01:00:51.240604 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.240531 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3f445db-61b7-4dcc-967d-4cf4d79ee302-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md\" (UID: \"e3f445db-61b7-4dcc-967d-4cf4d79ee302\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" Aug 04 01:00:51.240604 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.240568 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3f445db-61b7-4dcc-967d-4cf4d79ee302-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md\" (UID: \"e3f445db-61b7-4dcc-967d-4cf4d79ee302\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" Aug 04 01:00:51.241170 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.241144 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3f445db-61b7-4dcc-967d-4cf4d79ee302-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md\" (UID: \"e3f445db-61b7-4dcc-967d-4cf4d79ee302\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" Aug 04 01:00:51.242856 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.242837 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3f445db-61b7-4dcc-967d-4cf4d79ee302-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md\" (UID: \"e3f445db-61b7-4dcc-967d-4cf4d79ee302\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" Aug 04 01:00:51.247259 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.247236 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2xf4\" (UniqueName: \"kubernetes.io/projected/e3f445db-61b7-4dcc-967d-4cf4d79ee302-kube-api-access-x2xf4\") pod \"maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md\" (UID: \"e3f445db-61b7-4dcc-967d-4cf4d79ee302\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" Aug 04 01:00:51.317758 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.317709 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" Aug 04 01:00:51.447171 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.447139 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md"] Aug 04 01:00:51.450330 ip-10-0-138-102 kubenswrapper[2570]: W0804 01:00:51.450299 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f445db_61b7_4dcc_967d_4cf4d79ee302.slice/crio-14c41bf159a515b156b7ab409bc0296866f8fab0559259742c3772a6cda7620e WatchSource:0}: Error finding container 14c41bf159a515b156b7ab409bc0296866f8fab0559259742c3772a6cda7620e: Status 404 returned error can't find the container with id 14c41bf159a515b156b7ab409bc0296866f8fab0559259742c3772a6cda7620e Aug 04 01:00:51.635902 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.635832 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" event={"ID":"e3f445db-61b7-4dcc-967d-4cf4d79ee302","Type":"ContainerStarted","Data":"14c41bf159a515b156b7ab409bc0296866f8fab0559259742c3772a6cda7620e"} Aug 04 01:00:51.943362 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:51.943299 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:52.044149 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:52.044119 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:00:52.642463 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:52.642418 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" event={"ID":"e3f445db-61b7-4dcc-967d-4cf4d79ee302","Type":"ContainerStarted","Data":"492b3842b3ce35c6ba3ab803e8ba12f4ce8afd9e51a525e4b6f7a26080eee418"} Aug 04 01:00:52.642658 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:52.642493 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" Aug 04 01:00:52.659637 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:52.659587 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" podStartSLOduration=2.345945525 podStartE2EDuration="2.659571905s" totalImagesPullingTime="313.62638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 01:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:00:52.658062651 +0000 UTC m=+3499.140729009" watchObservedRunningTime="2026-08-04 01:00:52.659571905 +0000 UTC m=+3499.142238234" Aug 04 01:00:53.825090 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:53.825057 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 01:00:58.652746 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:00:58.652646 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" Aug 04 01:01:04.020981 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:04.020945 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:01:32.715701 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.715669 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg"] Aug 04 01:01:32.719570 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.719546 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.722619 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.722597 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0825a6\"/\"default-dockercfg-2lm5s\"" Aug 04 01:01:32.722718 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.722597 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0825a6\"/\"openshift-service-ca.crt\"" Aug 04 01:01:32.722718 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.722647 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0825a6\"/\"ipp-route-0825a6-kserve-self-signed-certs\"" Aug 04 01:01:32.722718 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.722648 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0825a6\"/\"kube-root-ca.crt\"" Aug 04 01:01:32.727276 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.727210 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg"] Aug 04 01:01:32.743948 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.743921 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:01:32.770021 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.769997 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-model-cache\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.770109 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.770030 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9765f094-7563-404f-9d73-8caf8f4dbf41-tls-certs\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.770109 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.770065 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-home\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.770179 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.770126 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbbsp\" (UniqueName: \"kubernetes.io/projected/9765f094-7563-404f-9d73-8caf8f4dbf41-kube-api-access-vbbsp\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.770215 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.770179 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-dshm\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.770215 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.770198 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-tmp-dir\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.871327 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.871304 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-dshm\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.871410 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.871333 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-tmp-dir\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.871474 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.871458 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-model-cache\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.871526 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.871498 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9765f094-7563-404f-9d73-8caf8f4dbf41-tls-certs\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.871576 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.871547 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-home\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.871576 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.871570 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbbsp\" (UniqueName: \"kubernetes.io/projected/9765f094-7563-404f-9d73-8caf8f4dbf41-kube-api-access-vbbsp\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.871754 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.871738 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-tmp-dir\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.871826 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.871802 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-model-cache\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.871922 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.871904 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-home\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.873646 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.873623 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-dshm\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.873880 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.873866 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9765f094-7563-404f-9d73-8caf8f4dbf41-tls-certs\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:32.880839 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:32.880819 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbbsp\" (UniqueName: \"kubernetes.io/projected/9765f094-7563-404f-9d73-8caf8f4dbf41-kube-api-access-vbbsp\") pod \"ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:33.030970 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:33.030950 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:33.157974 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:33.157948 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg"] Aug 04 01:01:33.159890 ip-10-0-138-102 kubenswrapper[2570]: W0804 01:01:33.159862 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9765f094_7563_404f_9d73_8caf8f4dbf41.slice/crio-08ea23fb6f8def68169dca3dae97355832eeb86c85e9539eed3c11c5ca540302 WatchSource:0}: Error finding container 08ea23fb6f8def68169dca3dae97355832eeb86c85e9539eed3c11c5ca540302: Status 404 returned error can't find the container with id 08ea23fb6f8def68169dca3dae97355832eeb86c85e9539eed3c11c5ca540302 Aug 04 01:01:33.854517 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:33.854475 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" event={"ID":"9765f094-7563-404f-9d73-8caf8f4dbf41","Type":"ContainerStarted","Data":"fc3b271aa6b5a848b94ff249512171f0a29415e688daba93636443fcc4743ed3"} Aug 04 01:01:33.854517 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:33.854515 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" event={"ID":"9765f094-7563-404f-9d73-8caf8f4dbf41","Type":"ContainerStarted","Data":"08ea23fb6f8def68169dca3dae97355832eeb86c85e9539eed3c11c5ca540302"} Aug 04 01:01:33.870989 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:33.870943 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" podStartSLOduration=1.870927574 podStartE2EDuration="1.870927574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 01:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:01:33.869386512 +0000 UTC m=+3540.352052841" watchObservedRunningTime="2026-08-04 01:01:33.870927574 +0000 UTC m=+3540.353593960" Aug 04 01:01:34.656733 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:34.656692 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:01:34.757128 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:34.757094 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:01:35.519708 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:35.519670 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 01:01:43.031741 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:43.031699 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:43.032304 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:43.031755 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:43.044649 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:43.044623 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:43.916917 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:43.916884 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:01:59.930515 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:01:59.930478 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 01:02:11.587824 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:11.587788 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:12.535879 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:12.535830 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:12.707533 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:12.707489 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:12.956265 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:12.956174 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:15.768374 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:15.768319 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:15.867615 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:15.867577 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:18.309594 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.309562 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs"] Aug 04 01:02:18.313013 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.312996 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" Aug 04 01:02:18.315156 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.315136 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-26288f-serving-cert\"" Aug 04 01:02:18.322173 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.322146 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs"] Aug 04 01:02:18.338371 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.338345 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a386647f-1852-4c40-82cb-9ccc9c27dae1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs\" (UID: \"a386647f-1852-4c40-82cb-9ccc9c27dae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" Aug 04 01:02:18.338466 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.338395 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbzgb\" (UniqueName: \"kubernetes.io/projected/a386647f-1852-4c40-82cb-9ccc9c27dae1-kube-api-access-wbzgb\") pod \"maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs\" (UID: \"a386647f-1852-4c40-82cb-9ccc9c27dae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" Aug 04 01:02:18.338466 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.338419 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a386647f-1852-4c40-82cb-9ccc9c27dae1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs\" (UID: \"a386647f-1852-4c40-82cb-9ccc9c27dae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" Aug 04 01:02:18.403280 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.403255 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:18.439775 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.439748 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzgb\" (UniqueName: \"kubernetes.io/projected/a386647f-1852-4c40-82cb-9ccc9c27dae1-kube-api-access-wbzgb\") pod \"maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs\" (UID: \"a386647f-1852-4c40-82cb-9ccc9c27dae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" Aug 04 01:02:18.439775 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.439777 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a386647f-1852-4c40-82cb-9ccc9c27dae1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs\" (UID: \"a386647f-1852-4c40-82cb-9ccc9c27dae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" Aug 04 01:02:18.439911 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.439848 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a386647f-1852-4c40-82cb-9ccc9c27dae1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs\" (UID: \"a386647f-1852-4c40-82cb-9ccc9c27dae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" Aug 04 01:02:18.439957 ip-10-0-138-102 kubenswrapper[2570]: E0804 01:02:18.439947 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-serving-cert: secret "maas-api-e2e-ipp-cleanup-26288f-serving-cert" not found Aug 04 01:02:18.440013 ip-10-0-138-102 kubenswrapper[2570]: E0804 01:02:18.440002 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a386647f-1852-4c40-82cb-9ccc9c27dae1-maas-api-tls podName:a386647f-1852-4c40-82cb-9ccc9c27dae1 nodeName:}" failed. No retries permitted until 2026-08-04 01:02:18.939984088 +0000 UTC m=+3585.422650397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a386647f-1852-4c40-82cb-9ccc9c27dae1-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" (UID: "a386647f-1852-4c40-82cb-9ccc9c27dae1") : secret "maas-api-e2e-ipp-cleanup-26288f-serving-cert" not found Aug 04 01:02:18.440581 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.440548 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a386647f-1852-4c40-82cb-9ccc9c27dae1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs\" (UID: \"a386647f-1852-4c40-82cb-9ccc9c27dae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" Aug 04 01:02:18.447876 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.447855 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbzgb\" (UniqueName: \"kubernetes.io/projected/a386647f-1852-4c40-82cb-9ccc9c27dae1-kube-api-access-wbzgb\") pod \"maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs\" (UID: \"a386647f-1852-4c40-82cb-9ccc9c27dae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" Aug 04 01:02:18.524120 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.524095 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:18.878182 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.878152 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:18.944412 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.944381 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a386647f-1852-4c40-82cb-9ccc9c27dae1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs\" (UID: \"a386647f-1852-4c40-82cb-9ccc9c27dae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" Aug 04 01:02:18.946733 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.946707 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a386647f-1852-4c40-82cb-9ccc9c27dae1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs\" (UID: \"a386647f-1852-4c40-82cb-9ccc9c27dae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" Aug 04 01:02:18.977413 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:18.977380 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:19.224260 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:19.224175 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" Aug 04 01:02:19.559720 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:19.559697 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs"] Aug 04 01:02:19.561646 ip-10-0-138-102 kubenswrapper[2570]: W0804 01:02:19.561618 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda386647f_1852_4c40_82cb_9ccc9c27dae1.slice/crio-f9ca35c76f65283b68e763aa1eb5f62eb221effc0385ba4dd9c835558bc1dc4d WatchSource:0}: Error finding container f9ca35c76f65283b68e763aa1eb5f62eb221effc0385ba4dd9c835558bc1dc4d: Status 404 returned error can't find the container with id f9ca35c76f65283b68e763aa1eb5f62eb221effc0385ba4dd9c835558bc1dc4d Aug 04 01:02:20.088411 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:20.088386 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" event={"ID":"a386647f-1852-4c40-82cb-9ccc9c27dae1","Type":"ContainerStarted","Data":"f9ca35c76f65283b68e763aa1eb5f62eb221effc0385ba4dd9c835558bc1dc4d"} Aug 04 01:02:21.095076 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:21.095026 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" event={"ID":"a386647f-1852-4c40-82cb-9ccc9c27dae1","Type":"ContainerStarted","Data":"880c2b53cf39e1494e8669613a2b60b5db01cb14f3e9e97746917f507c7233f8"} Aug 04 01:02:21.095501 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:21.095097 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" Aug 04 01:02:21.110599 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:21.110535 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" podStartSLOduration=2.587117167 podStartE2EDuration="3.110520324s" totalImagesPullingTime="523.403157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 01:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:02:21.109355355 +0000 UTC m=+3587.592021685" watchObservedRunningTime="2026-08-04 01:02:21.110520324 +0000 UTC m=+3587.593186644" Aug 04 01:02:22.840017 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:22.839984 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 01:02:27.105219 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:27.105142 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" Aug 04 01:02:30.479849 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:30.479788 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:34.297527 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:34.297493 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 01:02:34.300626 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:34.300606 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 01:02:34.306641 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:34.306617 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 01:02:34.318072 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:34.318054 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 01:02:34.320581 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:34.320565 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 01:02:34.325591 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:34.325575 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbj4t_c7975ad9-b029-4d54-9d07-f4ff8bdd6d57/ovn-acl-logging/0.log" Aug 04 01:02:48.767571 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:48.767534 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs"] Aug 04 01:02:48.768060 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:48.767788 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" podUID="a386647f-1852-4c40-82cb-9ccc9c27dae1" containerName="maas-api" containerID="cri-o://880c2b53cf39e1494e8669613a2b60b5db01cb14f3e9e97746917f507c7233f8" gracePeriod=30 Aug 04 01:02:48.797559 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:48.797527 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:48.869549 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:48.869521 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:48.924416 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:48.924390 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:48.988124 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:48.988092 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:49.007068 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.007045 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" Aug 04 01:02:49.099661 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.099604 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a386647f-1852-4c40-82cb-9ccc9c27dae1-openshift-service-ca\") pod \"a386647f-1852-4c40-82cb-9ccc9c27dae1\" (UID: \"a386647f-1852-4c40-82cb-9ccc9c27dae1\") " Aug 04 01:02:49.099770 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.099700 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a386647f-1852-4c40-82cb-9ccc9c27dae1-maas-api-tls\") pod \"a386647f-1852-4c40-82cb-9ccc9c27dae1\" (UID: \"a386647f-1852-4c40-82cb-9ccc9c27dae1\") " Aug 04 01:02:49.099770 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.099720 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbzgb\" (UniqueName: \"kubernetes.io/projected/a386647f-1852-4c40-82cb-9ccc9c27dae1-kube-api-access-wbzgb\") pod \"a386647f-1852-4c40-82cb-9ccc9c27dae1\" (UID: \"a386647f-1852-4c40-82cb-9ccc9c27dae1\") " Aug 04 01:02:49.099983 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.099961 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a386647f-1852-4c40-82cb-9ccc9c27dae1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a386647f-1852-4c40-82cb-9ccc9c27dae1" (UID: "a386647f-1852-4c40-82cb-9ccc9c27dae1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:02:49.101845 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.101819 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a386647f-1852-4c40-82cb-9ccc9c27dae1-kube-api-access-wbzgb" (OuterVolumeSpecName: "kube-api-access-wbzgb") pod "a386647f-1852-4c40-82cb-9ccc9c27dae1" (UID: "a386647f-1852-4c40-82cb-9ccc9c27dae1"). InnerVolumeSpecName "kube-api-access-wbzgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:02:49.101928 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.101842 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a386647f-1852-4c40-82cb-9ccc9c27dae1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a386647f-1852-4c40-82cb-9ccc9c27dae1" (UID: "a386647f-1852-4c40-82cb-9ccc9c27dae1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 01:02:49.201295 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.201272 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a386647f-1852-4c40-82cb-9ccc9c27dae1-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:02:49.201295 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.201293 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbzgb\" (UniqueName: \"kubernetes.io/projected/a386647f-1852-4c40-82cb-9ccc9c27dae1-kube-api-access-wbzgb\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:02:49.201425 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.201302 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a386647f-1852-4c40-82cb-9ccc9c27dae1-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:02:49.240898 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.240869 2570 generic.go:356] "Generic (PLEG): container finished" podID="a386647f-1852-4c40-82cb-9ccc9c27dae1" containerID="880c2b53cf39e1494e8669613a2b60b5db01cb14f3e9e97746917f507c7233f8" exitCode=0 Aug 04 01:02:49.241012 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.240917 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" Aug 04 01:02:49.241012 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.240926 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" event={"ID":"a386647f-1852-4c40-82cb-9ccc9c27dae1","Type":"ContainerDied","Data":"880c2b53cf39e1494e8669613a2b60b5db01cb14f3e9e97746917f507c7233f8"} Aug 04 01:02:49.241012 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.240964 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs" event={"ID":"a386647f-1852-4c40-82cb-9ccc9c27dae1","Type":"ContainerDied","Data":"f9ca35c76f65283b68e763aa1eb5f62eb221effc0385ba4dd9c835558bc1dc4d"} Aug 04 01:02:49.241012 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.240987 2570 scope.go:117] "RemoveContainer" containerID="880c2b53cf39e1494e8669613a2b60b5db01cb14f3e9e97746917f507c7233f8" Aug 04 01:02:49.250066 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.250046 2570 scope.go:117] "RemoveContainer" containerID="880c2b53cf39e1494e8669613a2b60b5db01cb14f3e9e97746917f507c7233f8" Aug 04 01:02:49.250315 ip-10-0-138-102 kubenswrapper[2570]: E0804 01:02:49.250297 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880c2b53cf39e1494e8669613a2b60b5db01cb14f3e9e97746917f507c7233f8\": container with ID starting with 880c2b53cf39e1494e8669613a2b60b5db01cb14f3e9e97746917f507c7233f8 not found: ID does not exist" containerID="880c2b53cf39e1494e8669613a2b60b5db01cb14f3e9e97746917f507c7233f8" Aug 04 01:02:49.250378 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.250323 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880c2b53cf39e1494e8669613a2b60b5db01cb14f3e9e97746917f507c7233f8"} err="failed to get container status \"880c2b53cf39e1494e8669613a2b60b5db01cb14f3e9e97746917f507c7233f8\": rpc error: code = NotFound desc = could not find container \"880c2b53cf39e1494e8669613a2b60b5db01cb14f3e9e97746917f507c7233f8\": container with ID starting with 880c2b53cf39e1494e8669613a2b60b5db01cb14f3e9e97746917f507c7233f8 not found: ID does not exist" Aug 04 01:02:49.262671 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.262651 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs"] Aug 04 01:02:49.268670 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.268646 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26288f-bf45c96d-7dkqs"] Aug 04 01:02:49.344759 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:49.344740 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:50.033487 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:50.033457 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a386647f-1852-4c40-82cb-9ccc9c27dae1" path="/var/lib/kubelet/pods/a386647f-1852-4c40-82cb-9ccc9c27dae1/volumes" Aug 04 01:02:51.823551 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:51.823514 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 01:02:55.653679 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:55.653643 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:55.756239 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:55.756191 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:55.980129 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:55.980036 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:02:56.158040 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:02:56.158012 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:03:11.820948 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:11.820913 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 01:03:16.535496 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:16.535446 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx"] Aug 04 01:03:16.536746 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:16.535704 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" podUID="a1520aab-8f0c-45e8-b598-aa68d9370b3e" containerName="maas-api" containerID="cri-o://e9083bad8f5c1794a2c699c59cf4764faf9e934862d4243e8e361e1cbd10829d" gracePeriod=30 Aug 04 01:03:16.579275 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:16.579247 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:03:16.673714 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:16.673657 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:03:16.767081 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:16.767054 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:03:16.786363 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:16.786311 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" Aug 04 01:03:16.922625 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:16.922585 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgptz\" (UniqueName: \"kubernetes.io/projected/a1520aab-8f0c-45e8-b598-aa68d9370b3e-kube-api-access-hgptz\") pod \"a1520aab-8f0c-45e8-b598-aa68d9370b3e\" (UID: \"a1520aab-8f0c-45e8-b598-aa68d9370b3e\") " Aug 04 01:03:16.922784 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:16.922671 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1520aab-8f0c-45e8-b598-aa68d9370b3e-openshift-service-ca\") pod \"a1520aab-8f0c-45e8-b598-aa68d9370b3e\" (UID: \"a1520aab-8f0c-45e8-b598-aa68d9370b3e\") " Aug 04 01:03:16.922784 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:16.922741 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1520aab-8f0c-45e8-b598-aa68d9370b3e-maas-api-tls\") pod \"a1520aab-8f0c-45e8-b598-aa68d9370b3e\" (UID: \"a1520aab-8f0c-45e8-b598-aa68d9370b3e\") " Aug 04 01:03:16.923149 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:16.923112 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1520aab-8f0c-45e8-b598-aa68d9370b3e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a1520aab-8f0c-45e8-b598-aa68d9370b3e" (UID: "a1520aab-8f0c-45e8-b598-aa68d9370b3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:03:16.924950 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:16.924922 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1520aab-8f0c-45e8-b598-aa68d9370b3e-kube-api-access-hgptz" (OuterVolumeSpecName: "kube-api-access-hgptz") pod "a1520aab-8f0c-45e8-b598-aa68d9370b3e" (UID: "a1520aab-8f0c-45e8-b598-aa68d9370b3e"). InnerVolumeSpecName "kube-api-access-hgptz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:03:16.925040 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:16.924951 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1520aab-8f0c-45e8-b598-aa68d9370b3e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a1520aab-8f0c-45e8-b598-aa68d9370b3e" (UID: "a1520aab-8f0c-45e8-b598-aa68d9370b3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 01:03:17.023828 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:17.023800 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1520aab-8f0c-45e8-b598-aa68d9370b3e-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:03:17.023828 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:17.023824 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1520aab-8f0c-45e8-b598-aa68d9370b3e-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:03:17.023953 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:17.023837 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgptz\" (UniqueName: \"kubernetes.io/projected/a1520aab-8f0c-45e8-b598-aa68d9370b3e-kube-api-access-hgptz\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:03:17.157282 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:17.157187 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:03:17.386827 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:17.386795 2570 generic.go:356] "Generic (PLEG): container finished" podID="a1520aab-8f0c-45e8-b598-aa68d9370b3e" containerID="e9083bad8f5c1794a2c699c59cf4764faf9e934862d4243e8e361e1cbd10829d" exitCode=0 Aug 04 01:03:17.386984 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:17.386845 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" Aug 04 01:03:17.386984 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:17.386881 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" event={"ID":"a1520aab-8f0c-45e8-b598-aa68d9370b3e","Type":"ContainerDied","Data":"e9083bad8f5c1794a2c699c59cf4764faf9e934862d4243e8e361e1cbd10829d"} Aug 04 01:03:17.386984 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:17.386933 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx" event={"ID":"a1520aab-8f0c-45e8-b598-aa68d9370b3e","Type":"ContainerDied","Data":"6ad968de5e753698a7822316b71cfcd42798e8130614c2edf88e07fedd9f1228"} Aug 04 01:03:17.386984 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:17.386948 2570 scope.go:117] "RemoveContainer" containerID="e9083bad8f5c1794a2c699c59cf4764faf9e934862d4243e8e361e1cbd10829d" Aug 04 01:03:17.396183 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:17.396164 2570 scope.go:117] "RemoveContainer" containerID="e9083bad8f5c1794a2c699c59cf4764faf9e934862d4243e8e361e1cbd10829d" Aug 04 01:03:17.396458 ip-10-0-138-102 kubenswrapper[2570]: E0804 01:03:17.396436 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9083bad8f5c1794a2c699c59cf4764faf9e934862d4243e8e361e1cbd10829d\": container with ID starting with e9083bad8f5c1794a2c699c59cf4764faf9e934862d4243e8e361e1cbd10829d not found: ID does not exist" containerID="e9083bad8f5c1794a2c699c59cf4764faf9e934862d4243e8e361e1cbd10829d" Aug 04 01:03:17.396554 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:17.396462 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9083bad8f5c1794a2c699c59cf4764faf9e934862d4243e8e361e1cbd10829d"} err="failed to get container status \"e9083bad8f5c1794a2c699c59cf4764faf9e934862d4243e8e361e1cbd10829d\": rpc error: code = NotFound desc = could not find container \"e9083bad8f5c1794a2c699c59cf4764faf9e934862d4243e8e361e1cbd10829d\": container with ID starting with e9083bad8f5c1794a2c699c59cf4764faf9e934862d4243e8e361e1cbd10829d not found: ID does not exist" Aug 04 01:03:17.407444 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:17.407397 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx"] Aug 04 01:03:17.413424 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:17.413404 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0825a6-d4bdd5dcf-6vmsx"] Aug 04 01:03:18.034243 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:18.034201 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1520aab-8f0c-45e8-b598-aa68d9370b3e" path="/var/lib/kubelet/pods/a1520aab-8f0c-45e8-b598-aa68d9370b3e/volumes" Aug 04 01:03:18.231685 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:18.231654 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 01:03:22.114871 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:22.114836 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg"] Aug 04 01:03:22.173872 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:22.173840 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:03:34.382760 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:34.382729 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:03:34.443772 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:34.443747 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:03:37.134054 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.133986 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" podUID="9765f094-7563-404f-9d73-8caf8f4dbf41" containerName="main" containerID="cri-o://fc3b271aa6b5a848b94ff249512171f0a29415e688daba93636443fcc4743ed3" gracePeriod=45 Aug 04 01:03:37.380776 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.380753 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:03:37.477521 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.477452 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9765f094-7563-404f-9d73-8caf8f4dbf41-tls-certs\") pod \"9765f094-7563-404f-9d73-8caf8f4dbf41\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " Aug 04 01:03:37.477521 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.477517 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-dshm\") pod \"9765f094-7563-404f-9d73-8caf8f4dbf41\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " Aug 04 01:03:37.477686 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.477542 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-tmp-dir\") pod \"9765f094-7563-404f-9d73-8caf8f4dbf41\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " Aug 04 01:03:37.477686 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.477563 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-model-cache\") pod \"9765f094-7563-404f-9d73-8caf8f4dbf41\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " Aug 04 01:03:37.477686 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.477622 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbbsp\" (UniqueName: \"kubernetes.io/projected/9765f094-7563-404f-9d73-8caf8f4dbf41-kube-api-access-vbbsp\") pod \"9765f094-7563-404f-9d73-8caf8f4dbf41\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " Aug 04 01:03:37.477847 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.477715 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-home\") pod \"9765f094-7563-404f-9d73-8caf8f4dbf41\" (UID: \"9765f094-7563-404f-9d73-8caf8f4dbf41\") " Aug 04 01:03:37.477847 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.477829 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-model-cache" (OuterVolumeSpecName: "model-cache") pod "9765f094-7563-404f-9d73-8caf8f4dbf41" (UID: "9765f094-7563-404f-9d73-8caf8f4dbf41"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:03:37.477963 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.477840 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9765f094-7563-404f-9d73-8caf8f4dbf41" (UID: "9765f094-7563-404f-9d73-8caf8f4dbf41"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:03:37.478090 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.478054 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-home" (OuterVolumeSpecName: "home") pod "9765f094-7563-404f-9d73-8caf8f4dbf41" (UID: "9765f094-7563-404f-9d73-8caf8f4dbf41"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:03:37.478090 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.478080 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-tmp-dir\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:03:37.478313 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.478102 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-model-cache\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:03:37.479618 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.479596 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-dshm" (OuterVolumeSpecName: "dshm") pod "9765f094-7563-404f-9d73-8caf8f4dbf41" (UID: "9765f094-7563-404f-9d73-8caf8f4dbf41"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 01:03:37.479618 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.479602 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9765f094-7563-404f-9d73-8caf8f4dbf41-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9765f094-7563-404f-9d73-8caf8f4dbf41" (UID: "9765f094-7563-404f-9d73-8caf8f4dbf41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 01:03:37.479748 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.479688 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9765f094-7563-404f-9d73-8caf8f4dbf41-kube-api-access-vbbsp" (OuterVolumeSpecName: "kube-api-access-vbbsp") pod "9765f094-7563-404f-9d73-8caf8f4dbf41" (UID: "9765f094-7563-404f-9d73-8caf8f4dbf41"). InnerVolumeSpecName "kube-api-access-vbbsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:03:37.489243 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.489195 2570 generic.go:356] "Generic (PLEG): container finished" podID="9765f094-7563-404f-9d73-8caf8f4dbf41" containerID="fc3b271aa6b5a848b94ff249512171f0a29415e688daba93636443fcc4743ed3" exitCode=0 Aug 04 01:03:37.489333 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.489263 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" Aug 04 01:03:37.489376 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.489259 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" event={"ID":"9765f094-7563-404f-9d73-8caf8f4dbf41","Type":"ContainerDied","Data":"fc3b271aa6b5a848b94ff249512171f0a29415e688daba93636443fcc4743ed3"} Aug 04 01:03:37.489376 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.489360 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg" event={"ID":"9765f094-7563-404f-9d73-8caf8f4dbf41","Type":"ContainerDied","Data":"08ea23fb6f8def68169dca3dae97355832eeb86c85e9539eed3c11c5ca540302"} Aug 04 01:03:37.489455 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.489377 2570 scope.go:117] "RemoveContainer" containerID="fc3b271aa6b5a848b94ff249512171f0a29415e688daba93636443fcc4743ed3" Aug 04 01:03:37.499887 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.499868 2570 scope.go:117] "RemoveContainer" containerID="fc3b271aa6b5a848b94ff249512171f0a29415e688daba93636443fcc4743ed3" Aug 04 01:03:37.500164 ip-10-0-138-102 kubenswrapper[2570]: E0804 01:03:37.500144 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3b271aa6b5a848b94ff249512171f0a29415e688daba93636443fcc4743ed3\": container with ID starting with fc3b271aa6b5a848b94ff249512171f0a29415e688daba93636443fcc4743ed3 not found: ID does not exist" containerID="fc3b271aa6b5a848b94ff249512171f0a29415e688daba93636443fcc4743ed3" Aug 04 01:03:37.500214 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.500173 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3b271aa6b5a848b94ff249512171f0a29415e688daba93636443fcc4743ed3"} err="failed to get container status \"fc3b271aa6b5a848b94ff249512171f0a29415e688daba93636443fcc4743ed3\": rpc error: code = NotFound desc = could not find container \"fc3b271aa6b5a848b94ff249512171f0a29415e688daba93636443fcc4743ed3\": container with ID starting with fc3b271aa6b5a848b94ff249512171f0a29415e688daba93636443fcc4743ed3 not found: ID does not exist" Aug 04 01:03:37.509580 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.509560 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg"] Aug 04 01:03:37.517113 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.517093 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-0825a6/ipp-route-0825a6-kserve-5db5cc5d8c-xzcgg"] Aug 04 01:03:37.578888 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.578864 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbbsp\" (UniqueName: \"kubernetes.io/projected/9765f094-7563-404f-9d73-8caf8f4dbf41-kube-api-access-vbbsp\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:03:37.578888 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.578885 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-home\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:03:37.579014 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.578895 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9765f094-7563-404f-9d73-8caf8f4dbf41-tls-certs\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:03:37.579014 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:37.578903 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9765f094-7563-404f-9d73-8caf8f4dbf41-dshm\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:03:38.034124 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:38.034091 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9765f094-7563-404f-9d73-8caf8f4dbf41" path="/var/lib/kubelet/pods/9765f094-7563-404f-9d73-8caf8f4dbf41/volumes" Aug 04 01:03:45.312574 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:45.312517 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:03:45.412302 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:45.412247 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:03:58.917379 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:58.917296 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:03:59.507118 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:03:59.507078 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:05.243066 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:05.243026 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:16.708108 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:16.708067 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:16.810680 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:16.810647 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:18.944791 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:18.944745 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:20.449815 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:20.449774 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:20.550137 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:20.550104 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:20.698797 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:20.698761 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:20.966778 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:20.966735 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:28.375597 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:28.375560 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md"] Aug 04 01:04:28.400158 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:28.400129 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:28.470898 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:28.470866 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:28.539564 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:28.539537 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:28.603775 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:28.603747 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:28.759883 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:28.759832 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" podUID="e3f445db-61b7-4dcc-967d-4cf4d79ee302" containerName="maas-api" containerID="cri-o://492b3842b3ce35c6ba3ab803e8ba12f4ce8afd9e51a525e4b6f7a26080eee418" gracePeriod=30 Aug 04 01:04:29.005509 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.005486 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" Aug 04 01:04:29.040960 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.040887 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:29.177106 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.177075 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3f445db-61b7-4dcc-967d-4cf4d79ee302-maas-api-tls\") pod \"e3f445db-61b7-4dcc-967d-4cf4d79ee302\" (UID: \"e3f445db-61b7-4dcc-967d-4cf4d79ee302\") " Aug 04 01:04:29.177258 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.177130 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2xf4\" (UniqueName: \"kubernetes.io/projected/e3f445db-61b7-4dcc-967d-4cf4d79ee302-kube-api-access-x2xf4\") pod \"e3f445db-61b7-4dcc-967d-4cf4d79ee302\" (UID: \"e3f445db-61b7-4dcc-967d-4cf4d79ee302\") " Aug 04 01:04:29.177258 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.177162 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3f445db-61b7-4dcc-967d-4cf4d79ee302-openshift-service-ca\") pod \"e3f445db-61b7-4dcc-967d-4cf4d79ee302\" (UID: \"e3f445db-61b7-4dcc-967d-4cf4d79ee302\") " Aug 04 01:04:29.177578 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.177555 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f445db-61b7-4dcc-967d-4cf4d79ee302-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e3f445db-61b7-4dcc-967d-4cf4d79ee302" (UID: "e3f445db-61b7-4dcc-967d-4cf4d79ee302"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:04:29.179140 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.179115 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3f445db-61b7-4dcc-967d-4cf4d79ee302-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e3f445db-61b7-4dcc-967d-4cf4d79ee302" (UID: "e3f445db-61b7-4dcc-967d-4cf4d79ee302"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 01:04:29.179259 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.179166 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f445db-61b7-4dcc-967d-4cf4d79ee302-kube-api-access-x2xf4" (OuterVolumeSpecName: "kube-api-access-x2xf4") pod "e3f445db-61b7-4dcc-967d-4cf4d79ee302" (UID: "e3f445db-61b7-4dcc-967d-4cf4d79ee302"). InnerVolumeSpecName "kube-api-access-x2xf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:04:29.278130 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.278104 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3f445db-61b7-4dcc-967d-4cf4d79ee302-maas-api-tls\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:04:29.278130 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.278129 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2xf4\" (UniqueName: \"kubernetes.io/projected/e3f445db-61b7-4dcc-967d-4cf4d79ee302-kube-api-access-x2xf4\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:04:29.278295 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.278139 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3f445db-61b7-4dcc-967d-4cf4d79ee302-openshift-service-ca\") on node \"ip-10-0-138-102.ec2.internal\" DevicePath \"\"" Aug 04 01:04:29.759509 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.759478 2570 generic.go:356] "Generic (PLEG): container finished" podID="e3f445db-61b7-4dcc-967d-4cf4d79ee302" containerID="492b3842b3ce35c6ba3ab803e8ba12f4ce8afd9e51a525e4b6f7a26080eee418" exitCode=0 Aug 04 01:04:29.759919 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.759524 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" Aug 04 01:04:29.759919 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.759528 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" event={"ID":"e3f445db-61b7-4dcc-967d-4cf4d79ee302","Type":"ContainerDied","Data":"492b3842b3ce35c6ba3ab803e8ba12f4ce8afd9e51a525e4b6f7a26080eee418"} Aug 04 01:04:29.759919 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.759636 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md" event={"ID":"e3f445db-61b7-4dcc-967d-4cf4d79ee302","Type":"ContainerDied","Data":"14c41bf159a515b156b7ab409bc0296866f8fab0559259742c3772a6cda7620e"} Aug 04 01:04:29.759919 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.759658 2570 scope.go:117] "RemoveContainer" containerID="492b3842b3ce35c6ba3ab803e8ba12f4ce8afd9e51a525e4b6f7a26080eee418" Aug 04 01:04:29.769184 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.769161 2570 scope.go:117] "RemoveContainer" containerID="492b3842b3ce35c6ba3ab803e8ba12f4ce8afd9e51a525e4b6f7a26080eee418" Aug 04 01:04:29.769491 ip-10-0-138-102 kubenswrapper[2570]: E0804 01:04:29.769470 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492b3842b3ce35c6ba3ab803e8ba12f4ce8afd9e51a525e4b6f7a26080eee418\": container with ID starting with 492b3842b3ce35c6ba3ab803e8ba12f4ce8afd9e51a525e4b6f7a26080eee418 not found: ID does not exist" containerID="492b3842b3ce35c6ba3ab803e8ba12f4ce8afd9e51a525e4b6f7a26080eee418" Aug 04 01:04:29.769561 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.769499 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492b3842b3ce35c6ba3ab803e8ba12f4ce8afd9e51a525e4b6f7a26080eee418"} err="failed to get container status \"492b3842b3ce35c6ba3ab803e8ba12f4ce8afd9e51a525e4b6f7a26080eee418\": rpc error: code = NotFound desc = could not find container \"492b3842b3ce35c6ba3ab803e8ba12f4ce8afd9e51a525e4b6f7a26080eee418\": container with ID starting with 492b3842b3ce35c6ba3ab803e8ba12f4ce8afd9e51a525e4b6f7a26080eee418 not found: ID does not exist" Aug 04 01:04:29.780476 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.780454 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md"] Aug 04 01:04:29.782285 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:29.782264 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbf8c-54549cbd87-n52md"] Aug 04 01:04:30.033380 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:30.033319 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3f445db-61b7-4dcc-967d-4cf4d79ee302" path="/var/lib/kubelet/pods/e3f445db-61b7-4dcc-967d-4cf4d79ee302/volumes" Aug 04 01:04:30.733336 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:30.733308 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 01:04:34.811385 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:34.811348 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:34.911472 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:34.911441 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:35.074525 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:35.074429 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:35.356312 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:35.356204 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:41.114511 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:41.114479 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:41.214464 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:41.214436 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:41.278278 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:41.278251 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:41.333978 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:41.333951 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:41.643600 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:41.643570 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:43.421253 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:43.421202 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 01:04:47.716286 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:47.716246 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:47.778330 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:47.778299 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:47.997821 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:47.997798 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:48.230520 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:48.230491 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:52.847489 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:52.847446 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:52.933614 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:52.933586 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:53.036037 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:53.036004 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:53.136792 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:53.136719 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:53.539542 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:53.539505 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:55.128059 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:55.128027 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tws8j"] Aug 04 01:04:58.761025 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:58.760986 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:59.448969 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:59.448937 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:59.549248 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:59.549189 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:04:59.725312 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:04:59.725214 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:05:00.027287 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:00.027254 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:05:00.464338 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:00.464268 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d65c755d-jvwcp_215639c2-cf65-4c3a-bc79-47bc79b10eb6/authorino/0.log" Aug 04 01:05:06.620430 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:06.620389 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67b98d597c-txvhq_846f9d3e-5acd-47df-8904-76a6ff0d0781/manager/0.log" Aug 04 01:05:06.625818 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:06.625795 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67b98d597c-txvhq_846f9d3e-5acd-47df-8904-76a6ff0d0781/copy-manifests/0.log" Aug 04 01:05:08.591586 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:08.591543 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d65c755d-jvwcp_215639c2-cf65-4c3a-bc79-47bc79b10eb6/authorino/0.log" Aug 04 01:05:08.950687 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:08.950607 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-b9hhd_7ae3da25-da2a-425e-b498-302e6198d7f3/kuadrant-console-plugin/0.log" Aug 04 01:05:09.215681 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:09.215611 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-tws8j_6a778ff6-d312-4fa6-9697-e71e03aef044/limitador/0.log" Aug 04 01:05:10.088328 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:10.088298 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-554b4579cb-m2ttw_f75898a0-3ed1-47e7-b54e-6ce840578c96/kube-auth-proxy/0.log" Aug 04 01:05:11.070105 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:11.070054 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-wzb7l_7c7bd60b-7ecb-4c68-82fd-dee5993b9c51/main/0.log" Aug 04 01:05:11.188420 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:11.188389 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-mzk7g_f48c767b-2ca5-440f-97ec-7ec88a52c9aa/main/0.log" Aug 04 01:05:17.112194 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:17.112149 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:05:17.212994 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:17.212962 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" limit=1024 Aug 04 01:05:19.816873 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:19.816838 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hrj5b_51f02f93-efb0-42f3-acaf-2de4e4db0cbc/global-pull-secret-syncer/0.log" Aug 04 01:05:19.957284 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:19.957255 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-28km9_7617213d-44e2-4227-bb9e-a9143683e61e/konnectivity-agent/0.log" Aug 04 01:05:20.115674 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:20.115594 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-102.ec2.internal_da457b2ccc5cb1e2a6488c095c0a0489/haproxy/0.log" Aug 04 01:05:25.478847 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:25.478764 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d65c755d-jvwcp_215639c2-cf65-4c3a-bc79-47bc79b10eb6/authorino/0.log" Aug 04 01:05:25.565077 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:25.565052 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-b9hhd_7ae3da25-da2a-425e-b498-302e6198d7f3/kuadrant-console-plugin/0.log" Aug 04 01:05:25.763022 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:25.763001 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-tws8j_6a778ff6-d312-4fa6-9697-e71e03aef044/limitador/0.log" Aug 04 01:05:27.360848 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:27.360823 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2621b7bb-8947-4bf3-af30-4817c5731b3f/alertmanager/0.log" Aug 04 01:05:27.382781 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:27.382756 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2621b7bb-8947-4bf3-af30-4817c5731b3f/config-reloader/0.log" Aug 04 01:05:27.403830 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:27.403776 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2621b7bb-8947-4bf3-af30-4817c5731b3f/kube-rbac-proxy-web/0.log" Aug 04 01:05:27.424454 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:27.424432 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2621b7bb-8947-4bf3-af30-4817c5731b3f/kube-rbac-proxy/0.log" Aug 04 01:05:27.450409 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:27.450391 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2621b7bb-8947-4bf3-af30-4817c5731b3f/kube-rbac-proxy-metric/0.log" Aug 04 01:05:27.473042 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:27.473019 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2621b7bb-8947-4bf3-af30-4817c5731b3f/prom-label-proxy/0.log" Aug 04 01:05:27.496904 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:27.496884 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2621b7bb-8947-4bf3-af30-4817c5731b3f/init-config-reloader/0.log" Aug 04 01:05:27.552917 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:27.552889 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/1.log" Aug 04 01:05:27.699411 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:27.699346 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbfsl_3021d0ff-dce6-45e6-a5f4-a0dae26dd3b5/cluster-monitoring-operator/0.log" Aug 04 01:05:28.085423 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:28.085396 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-svq5p_251d8954-f1bf-4a73-822b-784c13856b22/node-exporter/0.log" Aug 04 01:05:28.108489 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:28.108468 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-svq5p_251d8954-f1bf-4a73-822b-784c13856b22/kube-rbac-proxy/0.log" Aug 04 01:05:28.128836 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:28.128819 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-svq5p_251d8954-f1bf-4a73-822b-784c13856b22/init-textfile/0.log" Aug 04 01:05:28.240272 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:28.240248 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0245384-7186-41f9-a102-a48b908f6735/prometheus/0.log" Aug 04 01:05:28.260714 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:28.260697 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0245384-7186-41f9-a102-a48b908f6735/config-reloader/0.log" Aug 04 01:05:28.301621 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:28.301600 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0245384-7186-41f9-a102-a48b908f6735/thanos-sidecar/0.log" Aug 04 01:05:28.321685 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:28.321627 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0245384-7186-41f9-a102-a48b908f6735/kube-rbac-proxy-web/0.log" Aug 04 01:05:28.341042 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:28.341011 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0245384-7186-41f9-a102-a48b908f6735/kube-rbac-proxy/0.log" Aug 04 01:05:28.362986 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:28.362968 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0245384-7186-41f9-a102-a48b908f6735/kube-rbac-proxy-thanos/0.log" Aug 04 01:05:28.382604 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:28.382583 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0245384-7186-41f9-a102-a48b908f6735/init-config-reloader/0.log" Aug 04 01:05:28.413200 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:28.413081 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-4s9hh_aef52237-4118-485e-8785-8e182bd64689/prometheus-operator/0.log" Aug 04 01:05:28.428190 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:28.428173 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-4s9hh_aef52237-4118-485e-8785-8e182bd64689/kube-rbac-proxy/0.log" Aug 04 01:05:28.458580 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:28.458562 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-5r7jx_48432f55-35f4-4e33-909e-da66939c6d6c/prometheus-operator-admission-webhook/0.log" Aug 04 01:05:28.489045 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:28.489025 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5674c5f7bf-hdb6w_e22a206c-3881-4cff-8428-850d58b7b51d/telemeter-client/0.log" Aug 04 01:05:28.513525 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:28.513506 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5674c5f7bf-hdb6w_e22a206c-3881-4cff-8428-850d58b7b51d/reload/0.log" Aug 04 01:05:28.534337 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:28.534320 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5674c5f7bf-hdb6w_e22a206c-3881-4cff-8428-850d58b7b51d/kube-rbac-proxy/0.log" Aug 04 01:05:29.999563 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:29.999529 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p"] Aug 04 01:05:29.999957 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:29.999942 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9765f094-7563-404f-9d73-8caf8f4dbf41" containerName="main" Aug 04 01:05:30.000009 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:29.999961 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9765f094-7563-404f-9d73-8caf8f4dbf41" containerName="main" Aug 04 01:05:30.000009 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:29.999977 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1520aab-8f0c-45e8-b598-aa68d9370b3e" containerName="maas-api" Aug 04 01:05:30.000009 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:29.999985 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1520aab-8f0c-45e8-b598-aa68d9370b3e" containerName="maas-api" Aug 04 01:05:30.000009 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:29.999999 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f445db-61b7-4dcc-967d-4cf4d79ee302" containerName="maas-api" Aug 04 01:05:30.000009 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.000008 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f445db-61b7-4dcc-967d-4cf4d79ee302" containerName="maas-api" Aug 04 01:05:30.000240 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.000028 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a386647f-1852-4c40-82cb-9ccc9c27dae1" containerName="maas-api" Aug 04 01:05:30.000240 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.000036 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a386647f-1852-4c40-82cb-9ccc9c27dae1" containerName="maas-api" Aug 04 01:05:30.000240 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.000130 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f445db-61b7-4dcc-967d-4cf4d79ee302" containerName="maas-api" Aug 04 01:05:30.000240 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.000142 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9765f094-7563-404f-9d73-8caf8f4dbf41" containerName="main" Aug 04 01:05:30.000240 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.000151 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a386647f-1852-4c40-82cb-9ccc9c27dae1" containerName="maas-api" Aug 04 01:05:30.000240 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.000161 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1520aab-8f0c-45e8-b598-aa68d9370b3e" containerName="maas-api" Aug 04 01:05:30.003544 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.003530 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:30.005904 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.005884 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9cdcm\"/\"openshift-service-ca.crt\"" Aug 04 01:05:30.006011 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.005935 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9cdcm\"/\"default-dockercfg-wr28w\"" Aug 04 01:05:30.006831 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.006816 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9cdcm\"/\"kube-root-ca.crt\"" Aug 04 01:05:30.012886 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.012864 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p"] Aug 04 01:05:30.176260 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.176218 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f52ef15e-e88f-4d06-a1b7-43f49db06752-podres\") pod \"perf-node-gather-daemonset-9gs2p\" (UID: \"f52ef15e-e88f-4d06-a1b7-43f49db06752\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:30.176375 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.176273 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f52ef15e-e88f-4d06-a1b7-43f49db06752-lib-modules\") pod \"perf-node-gather-daemonset-9gs2p\" (UID: \"f52ef15e-e88f-4d06-a1b7-43f49db06752\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:30.176375 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.176313 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f52ef15e-e88f-4d06-a1b7-43f49db06752-proc\") pod \"perf-node-gather-daemonset-9gs2p\" (UID: \"f52ef15e-e88f-4d06-a1b7-43f49db06752\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:30.176375 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.176352 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htbvv\" (UniqueName: \"kubernetes.io/projected/f52ef15e-e88f-4d06-a1b7-43f49db06752-kube-api-access-htbvv\") pod \"perf-node-gather-daemonset-9gs2p\" (UID: \"f52ef15e-e88f-4d06-a1b7-43f49db06752\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:30.176483 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.176388 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f52ef15e-e88f-4d06-a1b7-43f49db06752-sys\") pod \"perf-node-gather-daemonset-9gs2p\" (UID: \"f52ef15e-e88f-4d06-a1b7-43f49db06752\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:30.277047 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.276989 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f52ef15e-e88f-4d06-a1b7-43f49db06752-sys\") pod \"perf-node-gather-daemonset-9gs2p\" (UID: \"f52ef15e-e88f-4d06-a1b7-43f49db06752\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:30.277047 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.277021 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f52ef15e-e88f-4d06-a1b7-43f49db06752-podres\") pod \"perf-node-gather-daemonset-9gs2p\" (UID: \"f52ef15e-e88f-4d06-a1b7-43f49db06752\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:30.277189 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.277052 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f52ef15e-e88f-4d06-a1b7-43f49db06752-lib-modules\") pod \"perf-node-gather-daemonset-9gs2p\" (UID: \"f52ef15e-e88f-4d06-a1b7-43f49db06752\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:30.277189 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.277074 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f52ef15e-e88f-4d06-a1b7-43f49db06752-proc\") pod \"perf-node-gather-daemonset-9gs2p\" (UID: \"f52ef15e-e88f-4d06-a1b7-43f49db06752\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:30.277189 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.277098 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f52ef15e-e88f-4d06-a1b7-43f49db06752-sys\") pod \"perf-node-gather-daemonset-9gs2p\" (UID: \"f52ef15e-e88f-4d06-a1b7-43f49db06752\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:30.277189 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.277116 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htbvv\" (UniqueName: \"kubernetes.io/projected/f52ef15e-e88f-4d06-a1b7-43f49db06752-kube-api-access-htbvv\") pod \"perf-node-gather-daemonset-9gs2p\" (UID: \"f52ef15e-e88f-4d06-a1b7-43f49db06752\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:30.277189 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.277142 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f52ef15e-e88f-4d06-a1b7-43f49db06752-proc\") pod \"perf-node-gather-daemonset-9gs2p\" (UID: \"f52ef15e-e88f-4d06-a1b7-43f49db06752\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:30.277189 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.277179 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f52ef15e-e88f-4d06-a1b7-43f49db06752-lib-modules\") pod \"perf-node-gather-daemonset-9gs2p\" (UID: \"f52ef15e-e88f-4d06-a1b7-43f49db06752\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:30.277434 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.277179 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f52ef15e-e88f-4d06-a1b7-43f49db06752-podres\") pod \"perf-node-gather-daemonset-9gs2p\" (UID: \"f52ef15e-e88f-4d06-a1b7-43f49db06752\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:30.288044 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.288016 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htbvv\" (UniqueName: \"kubernetes.io/projected/f52ef15e-e88f-4d06-a1b7-43f49db06752-kube-api-access-htbvv\") pod \"perf-node-gather-daemonset-9gs2p\" (UID: \"f52ef15e-e88f-4d06-a1b7-43f49db06752\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:30.313886 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.313867 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:30.405085 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.405054 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/1.log" Aug 04 01:05:30.417556 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.417523 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-mhqk6_036e01c1-b816-4ff5-8605-1ab0a192f4ee/console-operator/2.log" Aug 04 01:05:30.441406 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:30.441380 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p"] Aug 04 01:05:30.443268 ip-10-0-138-102 kubenswrapper[2570]: W0804 01:05:30.443242 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf52ef15e_e88f_4d06_a1b7_43f49db06752.slice/crio-54e5677e71fc40849e8a99ec150b41ecfd55f15b7339e58ac415f4907fdc5240 WatchSource:0}: Error finding container 54e5677e71fc40849e8a99ec150b41ecfd55f15b7339e58ac415f4907fdc5240: Status 404 returned error can't find the container with id 54e5677e71fc40849e8a99ec150b41ecfd55f15b7339e58ac415f4907fdc5240 Aug 04 01:05:31.072335 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:31.072296 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" event={"ID":"f52ef15e-e88f-4d06-a1b7-43f49db06752","Type":"ContainerStarted","Data":"9f87033eed872554a80f8597c3cf986716b1f8425221f86ef54f96ddf1e1a9bf"} Aug 04 01:05:31.072784 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:31.072347 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" event={"ID":"f52ef15e-e88f-4d06-a1b7-43f49db06752","Type":"ContainerStarted","Data":"54e5677e71fc40849e8a99ec150b41ecfd55f15b7339e58ac415f4907fdc5240"} Aug 04 01:05:31.072784 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:31.072379 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:31.089117 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:31.089068 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" podStartSLOduration=2.089053549 podStartE2EDuration="2.089053549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 01:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:05:31.087543778 +0000 UTC m=+3777.570210107" watchObservedRunningTime="2026-08-04 01:05:31.089053549 +0000 UTC m=+3777.571719879" Aug 04 01:05:31.612899 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:31.612864 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-clngv_628bb663-617f-4999-977b-a02ca26c1b19/volume-data-source-validator/0.log" Aug 04 01:05:32.620251 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:32.620203 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-88l8k_c3ee0892-91f7-4cab-8b96-829a660ab62d/dns/0.log" Aug 04 01:05:32.640546 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:32.640502 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-88l8k_c3ee0892-91f7-4cab-8b96-829a660ab62d/kube-rbac-proxy/0.log" Aug 04 01:05:32.728454 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:32.728432 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mbkx7_fc0ec9ec-6b13-46b6-9dc8-d1fdcfc8d117/dns-node-resolver/0.log" Aug 04 01:05:33.250314 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:33.250283 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29763360-vng42_2f998954-55dc-4e58-a70a-ad878047d419/image-pruner/0.log" Aug 04 01:05:33.334334 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:33.334309 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f5cfv_db57e300-ed1e-4192-9977-6fac6d117f66/node-ca/0.log" Aug 04 01:05:34.672134 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:34.672108 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-554b4579cb-m2ttw_f75898a0-3ed1-47e7-b54e-6ce840578c96/kube-auth-proxy/0.log" Aug 04 01:05:35.590492 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:35.590465 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fdgzw_df13c2f8-df07-47cb-82dc-2ac530b6985f/serve-healthcheck-canary/0.log" Aug 04 01:05:36.108681 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:36.108632 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-7b4h2_f076d5df-065e-4fa3-b3c2-341bd58b1c1d/insights-operator/0.log" Aug 04 01:05:36.110467 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:36.110438 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-7b4h2_f076d5df-065e-4fa3-b3c2-341bd58b1c1d/insights-operator/1.log" Aug 04 01:05:36.128773 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:36.128734 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cx6wf_74a07772-f95e-453a-8694-1ccce1cf23ac/kube-rbac-proxy/0.log" Aug 04 01:05:36.148927 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:36.148899 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cx6wf_74a07772-f95e-453a-8694-1ccce1cf23ac/exporter/0.log" Aug 04 01:05:36.168528 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:36.168503 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cx6wf_74a07772-f95e-453a-8694-1ccce1cf23ac/extractor/0.log" Aug 04 01:05:37.086464 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:37.086438 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-9gs2p" Aug 04 01:05:39.505196 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:39.505167 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67b98d597c-txvhq_846f9d3e-5acd-47df-8904-76a6ff0d0781/manager/0.log" Aug 04 01:05:39.527424 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:39.527399 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67b98d597c-txvhq_846f9d3e-5acd-47df-8904-76a6ff0d0781/copy-manifests/0.log" Aug 04 01:05:40.582611 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:40.582571 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-554b4579cb-m2ttw_f75898a0-3ed1-47e7-b54e-6ce840578c96/kube-auth-proxy/0.log" Aug 04 01:05:40.893009 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:40.892917 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7cfb669bf4-sd7rb_f8bdb018-ec6e-4409-a279-0703521c9882/manager/0.log" Aug 04 01:05:42.488144 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:42.488118 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-554b4579cb-m2ttw_f75898a0-3ed1-47e7-b54e-6ce840578c96/kube-auth-proxy/0.log" Aug 04 01:05:46.968217 ip-10-0-138-102 kubenswrapper[2570]: I0804 01:05:46.968180 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4jvtj_dbaf6a41-e197-4e72-bace-5b6621829d52/kube-multus/0.log"