Jul 15 19:21:31.424751 ip-10-0-141-225 systemd[1]: Starting Kubernetes Kubelet... Jul 15 19:21:31.781372 ip-10-0-141-225 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:21:31.781372 ip-10-0-141-225 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 19:21:31.781372 ip-10-0-141-225 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:21:31.781372 ip-10-0-141-225 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 19:21:31.781372 ip-10-0-141-225 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:21:31.783829 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.783740 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 19:21:31.788083 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788065 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:21:31.788123 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788084 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:21:31.788123 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788089 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:21:31.788123 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788092 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:21:31.788123 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788095 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:21:31.788123 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788098 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:21:31.788123 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788101 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:21:31.788123 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788104 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:21:31.788123 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788107 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:21:31.788123 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788110 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:21:31.788123 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788112 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:21:31.788123 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788115 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:21:31.788123 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788119 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:21:31.788123 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788122 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:21:31.788123 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788125 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:21:31.788123 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788128 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:21:31.788123 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788131 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788134 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788138 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788142 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788145 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788147 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788150 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788152 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788155 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788158 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788160 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788163 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788165 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788168 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788170 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788173 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788175 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788178 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788180 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788183 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:21:31.788522 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788185 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788187 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788190 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788192 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788194 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788197 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788199 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788201 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788204 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788206 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788209 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788211 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788214 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788217 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788220 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788223 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788228 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788231 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788233 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:21:31.789008 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788237 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788241 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788244 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788247 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788249 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788252 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788255 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788258 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788261 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788263 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788266 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788268 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788270 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788273 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788275 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788280 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788283 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788286 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788290 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:21:31.789480 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788293 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788296 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788300 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788302 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788305 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788308 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788311 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788314 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788316 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788319 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788322 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788325 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788327 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788331 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788333 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788336 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788338 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788341 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788343 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:21:31.790042 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788346 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788348 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788350 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788353 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788355 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.788357 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789086 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789093 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789096 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789099 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789101 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789104 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789106 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789109 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789112 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789114 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789117 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789119 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789122 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789124 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:21:31.790538 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789127 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789130 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789133 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789136 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789139 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789142 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789145 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789148 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789150 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789153 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789156 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789158 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789160 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789163 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789166 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789168 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789171 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789173 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789175 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789178 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:21:31.791040 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789180 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789182 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789185 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789188 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789190 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789193 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789195 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789197 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789200 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789204 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789209 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789212 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789216 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789219 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789221 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789225 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789227 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789230 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789233 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:21:31.791547 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789236 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789239 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789242 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789246 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789249 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789252 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789254 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789256 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789259 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789261 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789264 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789266 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789269 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789271 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789274 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789276 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789279 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789281 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789284 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789286 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:21:31.792009 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789288 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789291 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789293 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789296 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789300 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789303 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789305 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789308 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789311 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789313 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789316 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789319 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789322 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789326 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789330 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789333 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789335 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789338 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789341 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789343 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:21:31.792508 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789346 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789348 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789351 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789353 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789356 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.789359 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790519 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790526 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790530 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790533 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790536 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790539 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790541 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790544 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790547 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790550 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790556 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790559 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790562 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:21:31.792984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790565 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790568 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790570 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790573 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790575 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790579 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790582 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790585 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790588 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790592 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790595 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790598 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790601 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790604 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790607 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790610 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790613 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790616 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790619 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:21:31.793451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790621 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790624 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790626 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790629 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790632 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790634 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790637 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790640 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790642 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790645 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790648 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790651 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790654 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790656 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790659 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790661 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790664 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790666 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790669 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790672 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:21:31.793940 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790674 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790677 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790680 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790682 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790685 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790687 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790692 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790695 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790698 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790700 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790702 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790705 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790707 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790710 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790712 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790715 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790718 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790720 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790723 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:21:31.794436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790725 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790727 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790730 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790732 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790735 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790738 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790740 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790743 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790745 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790748 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790750 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790753 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790755 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790758 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790760 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790763 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790765 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790768 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790770 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790773 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:21:31.794942 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790775 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790777 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790780 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790782 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790786 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790790 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790793 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.790796 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790871 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790883 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790892 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790898 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790903 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790916 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790920 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790925 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790928 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790931 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790935 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790938 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790941 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790944 2564 flags.go:64] FLAG: --cgroup-root="" Jul 15 19:21:31.795412 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790947 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790950 2564 flags.go:64] FLAG: --client-ca-file="" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790953 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790955 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790961 2564 flags.go:64] FLAG: --cluster-domain="" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790964 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790967 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790970 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790973 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790977 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790980 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790983 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790986 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790989 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790992 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790995 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.790998 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791001 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791005 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791008 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791011 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791014 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791017 2564 flags.go:64] FLAG: --enable-server="true" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791020 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791024 2564 flags.go:64] FLAG: --event-burst="100" Jul 15 19:21:31.795987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791027 2564 flags.go:64] FLAG: --event-qps="50" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791030 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791033 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791035 2564 flags.go:64] FLAG: --eviction-hard="" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791039 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791042 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791045 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791050 2564 flags.go:64] FLAG: --eviction-soft="" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791054 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791056 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791059 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791062 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791065 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791068 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791071 2564 flags.go:64] FLAG: --feature-gates="" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791075 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791078 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791081 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791084 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791088 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791091 2564 flags.go:64] FLAG: --help="false" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791094 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-141-225.ec2.internal" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791097 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791100 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 19:21:31.796643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791103 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791106 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791109 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791112 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791115 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791117 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791120 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791123 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791126 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791129 2564 flags.go:64] FLAG: --kube-reserved="" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791132 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791135 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791138 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791140 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791143 2564 flags.go:64] FLAG: --lock-file="" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791147 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791150 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791153 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791159 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791162 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791164 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791168 2564 flags.go:64] FLAG: --logging-format="text" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791170 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791174 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 19:21:31.797246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791177 2564 flags.go:64] FLAG: --manifest-url="" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791179 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791186 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791191 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791195 2564 flags.go:64] FLAG: --max-pods="110" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791198 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791201 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791204 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791207 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791210 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791212 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791215 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791222 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791225 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791228 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791231 2564 flags.go:64] FLAG: --pod-cidr="" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791234 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791239 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791242 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791245 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791248 2564 flags.go:64] FLAG: --port="10250" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791250 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791253 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f92056b14a06e480" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791258 2564 flags.go:64] FLAG: --qos-reserved="" Jul 15 19:21:31.797863 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791261 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791263 2564 flags.go:64] FLAG: --register-node="true" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791266 2564 flags.go:64] FLAG: --register-with-taints="" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791269 2564 flags.go:64] FLAG: --registry-burst="10" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791273 2564 flags.go:64] FLAG: --registry-qps="5" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791275 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791278 2564 flags.go:64] FLAG: --reserved-memory="" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791282 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791285 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791288 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791290 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791293 2564 flags.go:64] FLAG: --runonce="false" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791298 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791301 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791303 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791306 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791309 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791311 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791314 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791317 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791320 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791323 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791325 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791328 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791331 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 19:21:31.798481 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791334 2564 flags.go:64] FLAG: --system-cgroups="" Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791336 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791341 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791344 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791347 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791351 2564 flags.go:64] FLAG: --tls-min-version="" Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791355 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791358 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791361 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791363 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791366 2564 flags.go:64] FLAG: --v="2" Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791370 2564 flags.go:64] FLAG: --version="false" Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791374 2564 flags.go:64] FLAG: --vmodule="" Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791378 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.791382 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791517 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791521 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791526 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791530 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791534 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791537 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791540 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:21:31.799149 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791543 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791545 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791548 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791550 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791553 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791555 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791558 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791561 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791563 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791566 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791569 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791571 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791574 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791576 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791579 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791581 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791585 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791587 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791590 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791592 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:21:31.799974 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791595 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791597 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791600 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791602 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791605 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791608 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791610 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791613 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791615 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791619 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791622 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791624 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791627 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791629 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791632 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791634 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791637 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791639 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791642 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791644 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:21:31.800598 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791646 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791649 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791652 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791654 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791657 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791659 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791662 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791664 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791667 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791670 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791673 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791675 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791677 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791680 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791682 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791685 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791687 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791690 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791693 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791696 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:21:31.801093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791698 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791702 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791705 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791707 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791710 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791712 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791715 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791717 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791720 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791722 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791724 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791727 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791729 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791732 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791735 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791737 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791740 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791743 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791747 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:21:31.801603 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791750 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:21:31.802065 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791753 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:21:31.802065 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791755 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:21:31.802065 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791758 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:21:31.802065 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791761 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:21:31.802065 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791764 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:21:31.802065 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791766 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:21:31.802065 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791769 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:21:31.802065 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791771 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:21:31.802065 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791774 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:21:31.802065 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791776 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:21:31.802065 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791780 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:21:31.802065 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.791783 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:21:31.802065 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.792617 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:21:31.802065 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.801141 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 19:21:31.802065 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.801156 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.801176 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.801185 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801243 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801248 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801251 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801254 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801257 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801260 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801263 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801266 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801269 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801272 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801275 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801277 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801280 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801282 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801285 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801288 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:21:31.802451 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801291 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801294 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801296 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801299 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801301 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801304 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801307 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801309 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801312 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801316 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801319 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801322 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801325 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801328 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801331 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801334 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801336 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801339 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801342 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801344 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:21:31.802920 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801347 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801349 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801352 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801354 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801357 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801359 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801363 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801366 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801369 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801371 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801375 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801378 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801381 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801383 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801386 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801388 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801390 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801393 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801395 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801398 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:21:31.803401 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801400 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801402 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801405 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801407 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801424 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801428 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801431 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801434 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801437 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801440 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801443 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801446 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801448 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801451 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801453 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801456 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801458 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801460 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801463 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801484 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:21:31.803895 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801489 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801492 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801495 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801499 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801502 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801506 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801510 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801513 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801515 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801518 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801521 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801523 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801526 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801528 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801531 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801533 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801536 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801538 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801541 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801543 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:21:31.804364 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801546 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:21:31.804875 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801549 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:21:31.804875 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801551 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:21:31.804875 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.801556 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:21:31.804875 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801673 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:21:31.804875 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801679 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:21:31.804875 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801682 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:21:31.804875 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801685 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:21:31.804875 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801688 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:21:31.804875 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801691 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:21:31.804875 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801693 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:21:31.804875 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801697 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:21:31.804875 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801701 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:21:31.804875 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801704 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:21:31.804875 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801707 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801711 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801714 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801717 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801720 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801722 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801725 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801728 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801730 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801732 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801735 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801737 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801740 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801742 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801744 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801747 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801750 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801752 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801755 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801757 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:21:31.805226 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801760 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801762 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801765 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801767 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801769 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801772 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801774 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801776 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801779 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801781 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801783 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801786 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801790 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801792 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801795 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801798 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801800 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801803 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801805 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801807 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:21:31.805725 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801809 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801812 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801814 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801817 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801819 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801821 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801823 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801826 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801829 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801831 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801834 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801836 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801839 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801841 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801844 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801846 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801849 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801852 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801854 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:21:31.806200 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801857 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801859 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801861 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801864 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801866 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801869 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801871 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801876 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801878 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801883 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801887 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801890 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801892 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801895 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801898 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801901 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801903 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801906 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801908 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:21:31.806741 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801910 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:21:31.807207 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801914 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:21:31.807207 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801917 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:21:31.807207 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801919 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:21:31.807207 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801922 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:21:31.807207 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801924 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:21:31.807207 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801927 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:21:31.807207 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801930 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:21:31.807207 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801932 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:21:31.807207 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801934 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:21:31.807207 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:31.801937 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:21:31.807207 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.801942 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:21:31.807207 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.802048 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 19:21:31.807207 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.804443 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 19:21:31.807207 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.805933 2564 server.go:1019] "Starting client certificate rotation" Jul 15 19:21:31.807569 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.806036 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 19:21:31.807569 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.806069 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 19:21:31.808125 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.808111 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 19:21:31.809312 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.809295 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 19:21:31.821300 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.821281 2564 log.go:25] "Validated CRI v1 runtime API" Jul 15 19:21:31.824785 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.824768 2564 log.go:25] "Validated CRI v1 image API" Jul 15 19:21:31.825612 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.825596 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 19:21:31.827964 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.827943 2564 fs.go:135] Filesystem UUIDs: map[3ac6fe90-ecd0-4b42-8bf9-4d28fc904517:/dev/nvme0n1p4 6c1317fd-932a-4cb9-b911-9eba0f91e606:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 15 19:21:31.828013 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.827964 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 19:21:31.833334 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.833314 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 19:21:31.834381 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.834257 2564 manager.go:216] Machine: {Timestamp:2026-07-15 19:21:31.832342985 +0000 UTC m=+0.303818265 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093907 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f7dcf0c76c0bce79b03360708a978 SystemUUID:ec2f7dcf-0c76-c0bc-e79b-03360708a978 BootID:07233f5f-c45f-4c59-97be-9f0ec4bd9477 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ef:ae:e5:82:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ef:ae:e5:82:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:e8:7f:d6:4f:17 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 19:21:31.834381 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.834371 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 19:21:31.834498 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.834463 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 19:21:31.838638 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.838419 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 19:21:31.838807 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.838639 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-225.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 19:21:31.839433 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.839424 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 19:21:31.839465 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.839437 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 19:21:31.839465 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.839444 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 19:21:31.840156 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.840145 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 19:21:31.841743 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.841732 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 19:21:31.845707 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.845696 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 15 19:21:31.845828 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.845819 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 19:21:31.845858 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.845852 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 15 19:21:31.845888 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.845865 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 19:21:31.845915 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.845887 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 19:21:31.845915 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.845901 2564 kubelet.go:398] "Adding apiserver pod source" Jul 15 19:21:31.845915 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.845913 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 19:21:31.846344 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.846332 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 19:21:31.846407 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.846351 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 19:21:31.848818 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.848798 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 19:21:31.850978 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.850964 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 19:21:31.851030 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.850995 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 19:21:31.851153 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.851143 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 19:21:31.851188 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.851158 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 19:21:31.851188 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.851164 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 19:21:31.851188 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.851170 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 19:21:31.851188 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.851178 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 19:21:31.851188 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.851183 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 19:21:31.851602 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.851592 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 19:21:31.851602 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.851601 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 19:21:31.851682 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.851608 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 19:21:31.851682 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.851614 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 19:21:31.851682 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.851621 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 19:21:31.851682 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.851629 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 19:21:31.852421 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.852410 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 19:21:31.852453 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.852423 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 19:21:31.855011 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.854993 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ch6ql" Jul 15 19:21:31.855217 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.855202 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-225.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 19:21:31.855217 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:31.855204 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 19:21:31.855307 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:31.855286 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-225.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 19:21:31.855642 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.855630 2564 server.go:1268] "Started kubelet" Jul 15 19:21:31.855746 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.855698 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 19:21:31.855800 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.855773 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 19:21:31.855844 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.855772 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 19:21:31.856617 ip-10-0-141-225 systemd[1]: Started Kubernetes Kubelet. Jul 15 19:21:31.858033 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.858006 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 15 19:21:31.859693 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.859678 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 19:21:31.860359 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.860347 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ch6ql" Jul 15 19:21:31.861288 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:31.861268 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 19:21:31.866071 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.866054 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 19:21:31.866148 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.866069 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 19:21:31.866148 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:31.866126 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-225.ec2.internal\" not found" Jul 15 19:21:31.866890 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.866828 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 19:21:31.866890 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.866827 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 19:21:31.866890 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.866849 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 19:21:31.867273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.866996 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 15 19:21:31.867273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.867006 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 15 19:21:31.867363 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.867354 2564 factory.go:55] Registering systemd factory Jul 15 19:21:31.867404 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.867372 2564 factory.go:223] Registration of the systemd container factory successfully Jul 15 19:21:31.867623 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.867610 2564 factory.go:153] Registering CRI-O factory Jul 15 19:21:31.867715 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.867625 2564 factory.go:223] Registration of the crio container factory successfully Jul 15 19:21:31.867715 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.867677 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 19:21:31.867715 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.867703 2564 factory.go:103] Registering Raw factory Jul 15 19:21:31.867715 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.867717 2564 manager.go:1196] Started watching for new ooms in manager Jul 15 19:21:31.868235 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.868222 2564 manager.go:349] Starting recovery of all containers Jul 15 19:21:31.868662 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.868642 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:21:31.871726 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:31.871685 2564 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-225.ec2.internal\" not found" node="ip-10-0-141-225.ec2.internal" Jul 15 19:21:31.881321 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.881185 2564 manager.go:354] Recovery completed Jul 15 19:21:31.885686 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.885664 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:21:31.887781 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.887762 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:21:31.887856 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.887794 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:21:31.887856 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.887805 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:21:31.888345 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.888329 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 19:21:31.888345 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.888340 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:21:31.888435 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.888347 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 19:21:31.888435 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.888368 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 15 19:21:31.889116 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.889101 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:21:31.889162 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.889129 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:21:31.889202 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.889139 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:21:31.889782 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.889766 2564 policy_none.go:49] "None policy: Start" Jul 15 19:21:31.889782 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.889783 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 19:21:31.889881 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.889794 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 19:21:31.891188 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.891177 2564 policy_none.go:47] "Start" Jul 15 19:21:31.928058 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.928036 2564 manager.go:337] "Starting Device Plugin manager" Jul 15 19:21:31.928165 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:31.928076 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 19:21:31.928165 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.928087 2564 server.go:87] "Starting device plugin registration server" Jul 15 19:21:31.928395 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.928381 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 19:21:31.928428 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.928397 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 19:21:31.928520 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.928505 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 19:21:31.928666 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.928582 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 19:21:31.928666 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:31.928591 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 19:21:31.929355 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:31.929330 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 19:21:31.929441 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:31.929384 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-225.ec2.internal\" not found" Jul 15 19:21:32.028973 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.028933 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:21:32.030235 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.030143 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:21:32.030235 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.030180 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:21:32.030235 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.030190 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:21:32.030235 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.030227 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.041122 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.041073 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.041122 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.041098 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-225.ec2.internal\": node \"ip-10-0-141-225.ec2.internal\" not found" Jul 15 19:21:32.051946 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.051908 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 19:21:32.053324 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.053297 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 19:21:32.053424 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.053346 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 19:21:32.053424 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.053370 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 19:21:32.053424 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.053409 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 19:21:32.056070 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.056053 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:21:32.058772 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.058751 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-225.ec2.internal\" not found" Jul 15 19:21:32.154079 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.154011 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-225.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal"] Jul 15 19:21:32.154243 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.154110 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:21:32.155115 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.155096 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:21:32.155206 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.155128 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:21:32.155206 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.155142 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:21:32.156608 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.156595 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:21:32.156780 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.156767 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.156819 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.156801 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:21:32.158865 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.158850 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-225.ec2.internal\" not found" Jul 15 19:21:32.164841 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.164826 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:21:32.164898 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.164839 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:21:32.164898 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.164855 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:21:32.164898 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.164860 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:21:32.164898 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.164865 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:21:32.164898 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.164869 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:21:32.166064 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.166050 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.166126 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.166076 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:21:32.166753 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.166738 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:21:32.166843 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.166763 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:21:32.166843 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.166779 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:21:32.167984 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.167969 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/90a23a491012cfceb92edd519600eade-config\") pod \"kube-apiserver-proxy-ip-10-0-141-225.ec2.internal\" (UID: \"90a23a491012cfceb92edd519600eade\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.168034 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.167991 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3083601b4042ee5819adc3570b16f90-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal\" (UID: \"e3083601b4042ee5819adc3570b16f90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.168034 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.168008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3083601b4042ee5819adc3570b16f90-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal\" (UID: \"e3083601b4042ee5819adc3570b16f90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.186027 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.186008 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-225.ec2.internal\" not found" node="ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.190256 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.190242 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-225.ec2.internal\" not found" node="ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.259901 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.259873 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-225.ec2.internal\" not found" Jul 15 19:21:32.268956 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.268935 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/90a23a491012cfceb92edd519600eade-config\") pod \"kube-apiserver-proxy-ip-10-0-141-225.ec2.internal\" (UID: \"90a23a491012cfceb92edd519600eade\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.269027 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.268961 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3083601b4042ee5819adc3570b16f90-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal\" (UID: \"e3083601b4042ee5819adc3570b16f90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.269027 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.268979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3083601b4042ee5819adc3570b16f90-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal\" (UID: \"e3083601b4042ee5819adc3570b16f90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.269027 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.269014 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3083601b4042ee5819adc3570b16f90-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal\" (UID: \"e3083601b4042ee5819adc3570b16f90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.269027 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.269023 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/90a23a491012cfceb92edd519600eade-config\") pod \"kube-apiserver-proxy-ip-10-0-141-225.ec2.internal\" (UID: \"90a23a491012cfceb92edd519600eade\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.269176 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.269054 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3083601b4042ee5819adc3570b16f90-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal\" (UID: \"e3083601b4042ee5819adc3570b16f90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.360846 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.360764 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-225.ec2.internal\" not found" Jul 15 19:21:32.461257 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.461224 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-225.ec2.internal\" not found" Jul 15 19:21:32.488795 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.488766 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.493940 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.493923 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.562144 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.562113 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-225.ec2.internal\" not found" Jul 15 19:21:32.662700 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.662618 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-225.ec2.internal\" not found" Jul 15 19:21:32.700285 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.700262 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:21:32.766591 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.766565 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.776988 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.776965 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 19:21:32.778456 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.778441 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal" Jul 15 19:21:32.788613 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.788595 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 19:21:32.805760 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.805733 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 19:21:32.805899 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.805864 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 19:21:32.805952 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.805896 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 19:21:32.805952 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.805896 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 19:21:32.847079 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.847057 2564 apiserver.go:52] "Watching apiserver" Jul 15 19:21:32.862290 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.862267 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 19:21:32.862621 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.862600 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-g6pfk","kube-system/kube-apiserver-proxy-ip-10-0-141-225.ec2.internal","openshift-cluster-node-tuning-operator/tuned-w24jl","openshift-dns/node-resolver-lrfwx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal","openshift-multus/multus-mwkrg","openshift-network-diagnostics/network-check-target-5w4r4","openshift-ovn-kubernetes/ovnkube-node-rr9n5","kube-system/global-pull-secret-syncer-pz2h5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj","openshift-image-registry/node-ca-n6mjl","openshift-multus/multus-additional-cni-plugins-5blhm","openshift-multus/network-metrics-daemon-zfvrb","openshift-network-operator/iptables-alerter-ddf8s"] Jul 15 19:21:32.864051 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.864035 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.865055 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.865038 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:32.865167 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.865105 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:21:32.865232 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.865182 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 19:16:32 +0000 UTC" deadline="2028-02-07 07:10:57.597714729 +0000 UTC" Jul 15 19:21:32.865232 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.865226 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13715h49m24.732492109s" Jul 15 19:21:32.865582 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.865562 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 19:21:32.865735 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.865720 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 19:21:32.865817 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.865763 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 19:21:32.865867 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.865821 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 19:21:32.866121 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.866109 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nwzbq\"" Jul 15 19:21:32.866175 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.866160 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 19:21:32.866742 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.866728 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g6pfk" Jul 15 19:21:32.867710 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.867695 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:32.867790 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.867757 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:21:32.868559 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.868542 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hmd7q\"" Jul 15 19:21:32.868610 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.868594 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 19:21:32.868679 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.868664 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 19:21:32.868883 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.868869 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lrfwx" Jul 15 19:21:32.870123 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.870108 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:32.870597 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.870585 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 19:21:32.870645 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.870620 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5s6xf\"" Jul 15 19:21:32.870688 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.870655 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 19:21:32.871206 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.871193 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.871728 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.871714 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:21:32.872001 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.871989 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 19:21:32.872114 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.872103 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cr4j7\"" Jul 15 19:21:32.872727 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.872714 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:32.872788 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.872770 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:21:32.872892 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.872877 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 19:21:32.872938 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.872906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-etc-kubernetes\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.872979 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.872933 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scxvz\" (UniqueName: \"kubernetes.io/projected/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-kube-api-access-scxvz\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.872979 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.872960 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-multus-socket-dir-parent\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.873047 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873003 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvvbn\" (UniqueName: \"kubernetes.io/projected/1a59a892-2494-489b-b5f2-038c7161cf7e-kube-api-access-kvvbn\") pod \"network-metrics-daemon-zfvrb\" (UID: \"1a59a892-2494-489b-b5f2-038c7161cf7e\") " pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:32.873047 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873025 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-cnibin\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.873121 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873047 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-run-netns\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.873121 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873071 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-multus-daemon-config\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.873121 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873095 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-run-multus-certs\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.873121 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873107 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 19:21:32.873292 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873120 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-system-cni-dir\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.873292 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873143 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-os-release\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.873292 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873164 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-cni-binary-copy\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.873292 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873167 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 19:21:32.873292 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-run-k8s-cni-cncf-io\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.873292 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873214 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-var-lib-cni-multus\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.873292 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873237 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-hostroot\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.873292 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873259 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-multus-conf-dir\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.873604 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873310 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 19:21:32.873604 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873312 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs\") pod \"network-metrics-daemon-zfvrb\" (UID: \"1a59a892-2494-489b-b5f2-038c7161cf7e\") " pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:32.873604 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873334 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2a3fc5a-d407-4ad9-9862-e5fcda33d18e-agent-certs\") pod \"konnectivity-agent-g6pfk\" (UID: \"d2a3fc5a-d407-4ad9-9862-e5fcda33d18e\") " pod="kube-system/konnectivity-agent-g6pfk" Jul 15 19:21:32.873604 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873365 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 19:21:32.873604 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873357 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kdzv\" (UniqueName: \"kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv\") pod \"network-check-target-5w4r4\" (UID: \"5d03ee98-6e6f-4ee1-8b78-d4da30c503ff\") " pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:32.873604 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873408 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 19:21:32.873604 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873455 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-multus-cni-dir\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.873604 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873517 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-var-lib-cni-bin\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.873604 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873540 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2a3fc5a-d407-4ad9-9862-e5fcda33d18e-konnectivity-ca\") pod \"konnectivity-agent-g6pfk\" (UID: \"d2a3fc5a-d407-4ad9-9862-e5fcda33d18e\") " pod="kube-system/konnectivity-agent-g6pfk" Jul 15 19:21:32.873604 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-var-lib-kubelet\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.873869 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.873613 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bmb8h\"" Jul 15 19:21:32.874117 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.874099 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:32.875545 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.875528 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n6mjl" Jul 15 19:21:32.875964 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.875950 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 19:21:32.876133 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.876111 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 19:21:32.876223 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.876208 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 19:21:32.876372 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.876355 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pf446\"" Jul 15 19:21:32.876945 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.876926 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:32.879898 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.877602 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 19:21:32.879898 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.877758 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 19:21:32.879898 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.877833 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 19:21:32.879898 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.877841 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cbmzq\"" Jul 15 19:21:32.879898 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.878815 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 19:21:32.879898 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.879038 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 19:21:32.879898 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.879088 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t7tdn\"" Jul 15 19:21:32.880846 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.880680 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ddf8s" Jul 15 19:21:32.882934 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.882914 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 19:21:32.883031 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.882949 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 19:21:32.883031 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.882953 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d8gdb\"" Jul 15 19:21:32.883132 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.883117 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:21:32.938437 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.938417 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 19:21:32.965241 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.965221 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-znbwv" Jul 15 19:21:32.967484 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.967453 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 19:21:32.973284 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.973261 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-znbwv" Jul 15 19:21:32.973917 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.973893 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-multus-socket-dir-parent\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.974033 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.973929 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvvbn\" (UniqueName: \"kubernetes.io/projected/1a59a892-2494-489b-b5f2-038c7161cf7e-kube-api-access-kvvbn\") pod \"network-metrics-daemon-zfvrb\" (UID: \"1a59a892-2494-489b-b5f2-038c7161cf7e\") " pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:32.974033 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974005 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-multus-socket-dir-parent\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.974139 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974056 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-multus-daemon-config\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.974139 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974098 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-run-multus-certs\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.974267 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974165 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-run-multus-certs\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.974267 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974209 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-node-log\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.974267 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974234 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-env-overrides\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.974444 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974267 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret\") pod \"global-pull-secret-syncer-pz2h5\" (UID: \"e089ab9b-564c-4a0d-9b14-4bce14a4f070\") " pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:32.974444 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-os-release\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.974444 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-var-lib-cni-multus\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.974444 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974370 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-multus-conf-dir\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.974444 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974404 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a323a5a4-1f3f-49b5-ae58-8b5c90592e5a-hosts-file\") pod \"node-resolver-lrfwx\" (UID: \"a323a5a4-1f3f-49b5-ae58-8b5c90592e5a\") " pod="openshift-dns/node-resolver-lrfwx" Jul 15 19:21:32.974444 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-device-dir\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:32.974444 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974433 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-var-lib-cni-multus\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.974444 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974432 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-os-release\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.974854 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974485 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-multus-conf-dir\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.974854 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974497 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-run\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:32.974854 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974556 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-lib-modules\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:32.974854 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974601 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-run-openvswitch\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.974854 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974640 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-ovnkube-config\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.974854 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974639 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-multus-daemon-config\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.974854 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-ovn-node-metrics-cert\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.974854 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974699 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-run-systemd\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.974854 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-cni-netd\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.974854 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974794 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:32.974854 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974841 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-registration-dir\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:32.974854 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-etc-kubernetes\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.975548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-cnibin\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.975548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974893 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-run-netns\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.975548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974917 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxkcn\" (UniqueName: \"kubernetes.io/projected/6432496f-c87d-403d-bdaf-f54af3038e6a-kube-api-access-fxkcn\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:32.975548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974929 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-etc-kubernetes\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.975548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-system-cni-dir\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.975548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974945 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-cnibin\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.975548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974966 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-run-netns\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.975548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974977 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-run-k8s-cni-cncf-io\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.975548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.974996 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-system-cni-dir\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.975548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975013 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-run-k8s-cni-cncf-io\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.975548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975014 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-sys\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:32.975548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975051 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.975548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975075 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-cni-binary-copy\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:32.975548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975100 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e089ab9b-564c-4a0d-9b14-4bce14a4f070-kubelet-config\") pod \"global-pull-secret-syncer-pz2h5\" (UID: \"e089ab9b-564c-4a0d-9b14-4bce14a4f070\") " pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:32.975548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975126 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2a3fc5a-d407-4ad9-9862-e5fcda33d18e-agent-certs\") pod \"konnectivity-agent-g6pfk\" (UID: \"d2a3fc5a-d407-4ad9-9862-e5fcda33d18e\") " pod="kube-system/konnectivity-agent-g6pfk" Jul 15 19:21:32.975548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975272 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kdzv\" (UniqueName: \"kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv\") pod \"network-check-target-5w4r4\" (UID: \"5d03ee98-6e6f-4ee1-8b78-d4da30c503ff\") " pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:32.975548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-modprobe-d\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:32.976414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975330 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-system-cni-dir\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:32.976414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975358 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a0dc43c-4fbe-4d98-aabe-831e75cda5ff-iptables-alerter-script\") pod \"iptables-alerter-ddf8s\" (UID: \"0a0dc43c-4fbe-4d98-aabe-831e75cda5ff\") " pod="openshift-network-operator/iptables-alerter-ddf8s" Jul 15 19:21:32.976414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-systemd\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:32.976414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975401 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-slash\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.976414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975418 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 19:21:32.976414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975427 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-cni-bin\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.976414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975488 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwjqv\" (UniqueName: \"kubernetes.io/projected/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-kube-api-access-rwjqv\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.976414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975540 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:32.976414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975556 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a0dc43c-4fbe-4d98-aabe-831e75cda5ff-host-slash\") pod \"iptables-alerter-ddf8s\" (UID: \"0a0dc43c-4fbe-4d98-aabe-831e75cda5ff\") " pod="openshift-network-operator/iptables-alerter-ddf8s" Jul 15 19:21:32.976414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975575 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e089ab9b-564c-4a0d-9b14-4bce14a4f070-dbus\") pod \"global-pull-secret-syncer-pz2h5\" (UID: \"e089ab9b-564c-4a0d-9b14-4bce14a4f070\") " pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:32.976414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-hostroot\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.976414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975622 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-run-netns\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.976414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975637 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:32.976414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975651 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sbz5\" (UniqueName: \"kubernetes.io/projected/98333cd6-ba91-4b38-bf05-64e47d2f0251-kube-api-access-7sbz5\") pod \"node-ca-n6mjl\" (UID: \"98333cd6-ba91-4b38-bf05-64e47d2f0251\") " pod="openshift-image-registry/node-ca-n6mjl" Jul 15 19:21:32.976414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975668 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs\") pod \"network-metrics-daemon-zfvrb\" (UID: \"1a59a892-2494-489b-b5f2-038c7161cf7e\") " pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:32.976414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975699 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-multus-cni-dir\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.976414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975730 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcnwb\" (UniqueName: \"kubernetes.io/projected/a323a5a4-1f3f-49b5-ae58-8b5c90592e5a-kube-api-access-xcnwb\") pod \"node-resolver-lrfwx\" (UID: \"a323a5a4-1f3f-49b5-ae58-8b5c90592e5a\") " pod="openshift-dns/node-resolver-lrfwx" Jul 15 19:21:32.977385 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.975753 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:21:32.977385 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975755 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-kubernetes\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:32.977385 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975778 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-etc-openvswitch\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.977385 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975802 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.977385 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.975833 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs podName:1a59a892-2494-489b-b5f2-038c7161cf7e nodeName:}" failed. No retries permitted until 2026-07-15 19:21:33.475801639 +0000 UTC m=+1.947276923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs") pod "network-metrics-daemon-zfvrb" (UID: "1a59a892-2494-489b-b5f2-038c7161cf7e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:21:32.977385 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975851 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-hostroot\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.977385 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975905 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-multus-cni-dir\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.977385 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-cnibin\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:32.977385 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.975989 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:32.977385 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976018 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-var-lib-kubelet\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.977385 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scxvz\" (UniqueName: \"kubernetes.io/projected/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-kube-api-access-scxvz\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.977385 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976063 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-sysctl-conf\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:32.977385 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976085 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-kubelet\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.977385 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976103 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-var-lib-kubelet\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.977385 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976108 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/98333cd6-ba91-4b38-bf05-64e47d2f0251-host\") pod \"node-ca-n6mjl\" (UID: \"98333cd6-ba91-4b38-bf05-64e47d2f0251\") " pod="openshift-image-registry/node-ca-n6mjl" Jul 15 19:21:32.977385 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976153 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rqvz\" (UniqueName: \"kubernetes.io/projected/0a0dc43c-4fbe-4d98-aabe-831e75cda5ff-kube-api-access-2rqvz\") pod \"iptables-alerter-ddf8s\" (UID: \"0a0dc43c-4fbe-4d98-aabe-831e75cda5ff\") " pod="openshift-network-operator/iptables-alerter-ddf8s" Jul 15 19:21:32.977385 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976174 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-sysconfig\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:32.978091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976194 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fac37a7e-282e-4f8f-b1b1-1c259d141c58-tmp\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:32.978091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976215 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-run-ovn\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.978091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976237 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-log-socket\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.978091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976263 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:32.978091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976305 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-sys-fs\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:32.978091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/98333cd6-ba91-4b38-bf05-64e47d2f0251-serviceca\") pod \"node-ca-n6mjl\" (UID: \"98333cd6-ba91-4b38-bf05-64e47d2f0251\") " pod="openshift-image-registry/node-ca-n6mjl" Jul 15 19:21:32.978091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976351 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-cni-binary-copy\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.978091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976374 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a323a5a4-1f3f-49b5-ae58-8b5c90592e5a-tmp-dir\") pod \"node-resolver-lrfwx\" (UID: \"a323a5a4-1f3f-49b5-ae58-8b5c90592e5a\") " pod="openshift-dns/node-resolver-lrfwx" Jul 15 19:21:32.978091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976401 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-sysctl-d\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:32.978091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-tuned\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:32.978091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976444 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-systemd-units\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.978091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976485 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-var-lib-openvswitch\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.978091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976512 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwx4t\" (UniqueName: \"kubernetes.io/projected/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-kube-api-access-wwx4t\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:32.978091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976545 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-var-lib-cni-bin\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.978091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976588 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-host\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:32.978091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976604 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-os-release\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:32.978091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976625 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-socket-dir\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:32.978917 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976651 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2a3fc5a-d407-4ad9-9862-e5fcda33d18e-konnectivity-ca\") pod \"konnectivity-agent-g6pfk\" (UID: \"d2a3fc5a-d407-4ad9-9862-e5fcda33d18e\") " pod="kube-system/konnectivity-agent-g6pfk" Jul 15 19:21:32.978917 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976660 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-host-var-lib-cni-bin\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.978917 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976675 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-var-lib-kubelet\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:32.978917 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976702 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwhv2\" (UniqueName: \"kubernetes.io/projected/fac37a7e-282e-4f8f-b1b1-1c259d141c58-kube-api-access-lwhv2\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:32.978917 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976754 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-ovnkube-script-lib\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:32.978917 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.976899 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-cni-binary-copy\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:32.978917 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.977028 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:21:32.978917 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.977178 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2a3fc5a-d407-4ad9-9862-e5fcda33d18e-konnectivity-ca\") pod \"konnectivity-agent-g6pfk\" (UID: \"d2a3fc5a-d407-4ad9-9862-e5fcda33d18e\") " pod="kube-system/konnectivity-agent-g6pfk" Jul 15 19:21:32.981575 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.980998 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2a3fc5a-d407-4ad9-9862-e5fcda33d18e-agent-certs\") pod \"konnectivity-agent-g6pfk\" (UID: \"d2a3fc5a-d407-4ad9-9862-e5fcda33d18e\") " pod="kube-system/konnectivity-agent-g6pfk" Jul 15 19:21:32.982985 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.982953 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:21:32.982985 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.982981 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:21:32.983141 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.982996 2564 projected.go:196] Error preparing data for projected volume kube-api-access-5kdzv for pod openshift-network-diagnostics/network-check-target-5w4r4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:21:32.983141 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:32.983061 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv podName:5d03ee98-6e6f-4ee1-8b78-d4da30c503ff nodeName:}" failed. No retries permitted until 2026-07-15 19:21:33.483044507 +0000 UTC m=+1.954519781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5kdzv" (UniqueName: "kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv") pod "network-check-target-5w4r4" (UID: "5d03ee98-6e6f-4ee1-8b78-d4da30c503ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:21:32.983295 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.983273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvvbn\" (UniqueName: \"kubernetes.io/projected/1a59a892-2494-489b-b5f2-038c7161cf7e-kube-api-access-kvvbn\") pod \"network-metrics-daemon-zfvrb\" (UID: \"1a59a892-2494-489b-b5f2-038c7161cf7e\") " pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:32.984963 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:32.984943 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scxvz\" (UniqueName: \"kubernetes.io/projected/004b9afd-c0c0-48b7-8b25-f3199c2af4a2-kube-api-access-scxvz\") pod \"multus-mwkrg\" (UID: \"004b9afd-c0c0-48b7-8b25-f3199c2af4a2\") " pod="openshift-multus/multus-mwkrg" Jul 15 19:21:33.053776 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:33.053750 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a23a491012cfceb92edd519600eade.slice/crio-83630cceddcb70cdbdc998fdda73a3e5555643fa5ed6b818b281eed6ecc3b87c WatchSource:0}: Error finding container 83630cceddcb70cdbdc998fdda73a3e5555643fa5ed6b818b281eed6ecc3b87c: Status 404 returned error can't find the container with id 83630cceddcb70cdbdc998fdda73a3e5555643fa5ed6b818b281eed6ecc3b87c Jul 15 19:21:33.053949 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:33.053937 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3083601b4042ee5819adc3570b16f90.slice/crio-5c7eef9260259815c12ea6673b17b46cf63fe69b4fa9c9170e8d802652bb25f1 WatchSource:0}: Error finding container 5c7eef9260259815c12ea6673b17b46cf63fe69b4fa9c9170e8d802652bb25f1: Status 404 returned error can't find the container with id 5c7eef9260259815c12ea6673b17b46cf63fe69b4fa9c9170e8d802652bb25f1 Jul 15 19:21:33.057839 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.057820 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:21:33.077948 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.077917 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-modprobe-d\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.078066 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.077953 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-system-cni-dir\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:33.078066 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.077982 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a0dc43c-4fbe-4d98-aabe-831e75cda5ff-iptables-alerter-script\") pod \"iptables-alerter-ddf8s\" (UID: \"0a0dc43c-4fbe-4d98-aabe-831e75cda5ff\") " pod="openshift-network-operator/iptables-alerter-ddf8s" Jul 15 19:21:33.078066 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078007 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-systemd\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.078066 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078032 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-slash\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.078066 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078036 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-system-cni-dir\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:33.078066 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078056 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-cni-bin\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.078438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078074 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-modprobe-d\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.078438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078093 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-systemd\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.078438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078081 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwjqv\" (UniqueName: \"kubernetes.io/projected/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-kube-api-access-rwjqv\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.078438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078126 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-slash\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.078438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078139 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-cni-bin\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.078438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078144 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:33.078438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a0dc43c-4fbe-4d98-aabe-831e75cda5ff-host-slash\") pod \"iptables-alerter-ddf8s\" (UID: \"0a0dc43c-4fbe-4d98-aabe-831e75cda5ff\") " pod="openshift-network-operator/iptables-alerter-ddf8s" Jul 15 19:21:33.078438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e089ab9b-564c-4a0d-9b14-4bce14a4f070-dbus\") pod \"global-pull-secret-syncer-pz2h5\" (UID: \"e089ab9b-564c-4a0d-9b14-4bce14a4f070\") " pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:33.078438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078211 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-run-netns\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.078438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078211 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:33.078438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:33.078438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sbz5\" (UniqueName: \"kubernetes.io/projected/98333cd6-ba91-4b38-bf05-64e47d2f0251-kube-api-access-7sbz5\") pod \"node-ca-n6mjl\" (UID: \"98333cd6-ba91-4b38-bf05-64e47d2f0251\") " pod="openshift-image-registry/node-ca-n6mjl" Jul 15 19:21:33.078438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078257 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-run-netns\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.078438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078279 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcnwb\" (UniqueName: \"kubernetes.io/projected/a323a5a4-1f3f-49b5-ae58-8b5c90592e5a-kube-api-access-xcnwb\") pod \"node-resolver-lrfwx\" (UID: \"a323a5a4-1f3f-49b5-ae58-8b5c90592e5a\") " pod="openshift-dns/node-resolver-lrfwx" Jul 15 19:21:33.078438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078289 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a0dc43c-4fbe-4d98-aabe-831e75cda5ff-host-slash\") pod \"iptables-alerter-ddf8s\" (UID: \"0a0dc43c-4fbe-4d98-aabe-831e75cda5ff\") " pod="openshift-network-operator/iptables-alerter-ddf8s" Jul 15 19:21:33.078438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-kubernetes\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.078438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078317 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e089ab9b-564c-4a0d-9b14-4bce14a4f070-dbus\") pod \"global-pull-secret-syncer-pz2h5\" (UID: \"e089ab9b-564c-4a0d-9b14-4bce14a4f070\") " pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:33.079348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078326 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:33.079348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078341 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-kubernetes\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.079348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-etc-openvswitch\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.079348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.079348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078405 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-cnibin\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:33.079348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078428 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-etc-openvswitch\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.079348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078439 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.079348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078436 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:33.079348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078501 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-cnibin\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:33.079348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078544 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-sysctl-conf\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.079348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078569 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a0dc43c-4fbe-4d98-aabe-831e75cda5ff-iptables-alerter-script\") pod \"iptables-alerter-ddf8s\" (UID: \"0a0dc43c-4fbe-4d98-aabe-831e75cda5ff\") " pod="openshift-network-operator/iptables-alerter-ddf8s" Jul 15 19:21:33.079348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-kubelet\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.079348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078612 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-kubelet\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.079348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078625 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/98333cd6-ba91-4b38-bf05-64e47d2f0251-host\") pod \"node-ca-n6mjl\" (UID: \"98333cd6-ba91-4b38-bf05-64e47d2f0251\") " pod="openshift-image-registry/node-ca-n6mjl" Jul 15 19:21:33.079348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078652 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rqvz\" (UniqueName: \"kubernetes.io/projected/0a0dc43c-4fbe-4d98-aabe-831e75cda5ff-kube-api-access-2rqvz\") pod \"iptables-alerter-ddf8s\" (UID: \"0a0dc43c-4fbe-4d98-aabe-831e75cda5ff\") " pod="openshift-network-operator/iptables-alerter-ddf8s" Jul 15 19:21:33.079348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078676 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-sysctl-conf\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.079348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078677 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-sysconfig\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.080273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078700 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/98333cd6-ba91-4b38-bf05-64e47d2f0251-host\") pod \"node-ca-n6mjl\" (UID: \"98333cd6-ba91-4b38-bf05-64e47d2f0251\") " pod="openshift-image-registry/node-ca-n6mjl" Jul 15 19:21:33.080273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078705 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fac37a7e-282e-4f8f-b1b1-1c259d141c58-tmp\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.080273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078722 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-sysconfig\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.080273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-run-ovn\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.080273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-log-socket\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.080273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:33.080273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-run-ovn\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.080273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-sys-fs\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:33.080273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078840 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-log-socket\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.080273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078846 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/98333cd6-ba91-4b38-bf05-64e47d2f0251-serviceca\") pod \"node-ca-n6mjl\" (UID: \"98333cd6-ba91-4b38-bf05-64e47d2f0251\") " pod="openshift-image-registry/node-ca-n6mjl" Jul 15 19:21:33.080273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078883 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a323a5a4-1f3f-49b5-ae58-8b5c90592e5a-tmp-dir\") pod \"node-resolver-lrfwx\" (UID: \"a323a5a4-1f3f-49b5-ae58-8b5c90592e5a\") " pod="openshift-dns/node-resolver-lrfwx" Jul 15 19:21:33.080273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078892 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-sys-fs\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:33.080273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-sysctl-d\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.080273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078935 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-tuned\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.080273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078961 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-systemd-units\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.080273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.078987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-var-lib-openvswitch\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.080273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079002 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:33.081130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079019 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwx4t\" (UniqueName: \"kubernetes.io/projected/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-kube-api-access-wwx4t\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:33.081130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079057 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-host\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.081130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079065 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-systemd-units\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.081130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079080 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-os-release\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:33.081130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079105 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-socket-dir\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:33.081130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-var-lib-kubelet\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.081130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079160 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwhv2\" (UniqueName: \"kubernetes.io/projected/fac37a7e-282e-4f8f-b1b1-1c259d141c58-kube-api-access-lwhv2\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.081130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-ovnkube-script-lib\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.081130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079200 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a323a5a4-1f3f-49b5-ae58-8b5c90592e5a-tmp-dir\") pod \"node-resolver-lrfwx\" (UID: \"a323a5a4-1f3f-49b5-ae58-8b5c90592e5a\") " pod="openshift-dns/node-resolver-lrfwx" Jul 15 19:21:33.081130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-node-log\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.081130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079245 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-env-overrides\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.081130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079271 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-var-lib-openvswitch\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.081130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079270 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:33.081130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079082 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-sysctl-d\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.081130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079322 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret\") pod \"global-pull-secret-syncer-pz2h5\" (UID: \"e089ab9b-564c-4a0d-9b14-4bce14a4f070\") " pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:33.081130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079350 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-node-log\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.081130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079353 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-socket-dir\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:33.081718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079373 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-os-release\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:33.081718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079546 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-host\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.081718 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:33.079631 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:21:33.081718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a323a5a4-1f3f-49b5-ae58-8b5c90592e5a-hosts-file\") pod \"node-resolver-lrfwx\" (UID: \"a323a5a4-1f3f-49b5-ae58-8b5c90592e5a\") " pod="openshift-dns/node-resolver-lrfwx" Jul 15 19:21:33.081718 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:33.079700 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret podName:e089ab9b-564c-4a0d-9b14-4bce14a4f070 nodeName:}" failed. No retries permitted until 2026-07-15 19:21:33.579680393 +0000 UTC m=+2.051155661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret") pod "global-pull-secret-syncer-pz2h5" (UID: "e089ab9b-564c-4a0d-9b14-4bce14a4f070") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:21:33.081718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079763 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-device-dir\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:33.081718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079791 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-run\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.081718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079817 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-lib-modules\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.081718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079843 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-ovnkube-script-lib\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.081718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079843 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-run-openvswitch\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.081718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079893 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-run\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.081718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-ovnkube-config\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.081718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079903 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-var-lib-kubelet\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.081718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079925 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-ovn-node-metrics-cert\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.081718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-run-systemd\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.081718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079973 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-cni-netd\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.081718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.079990 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:33.082327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-registration-dir\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:33.082327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080035 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-lib-modules\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.082327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080044 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxkcn\" (UniqueName: \"kubernetes.io/projected/6432496f-c87d-403d-bdaf-f54af3038e6a-kube-api-access-fxkcn\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:33.082327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080062 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-sys\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.082327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080082 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.082327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080086 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-cni-netd\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.082327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080137 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.082327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080145 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-registration-dir\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:33.082327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080159 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-cni-binary-copy\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:33.082327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e089ab9b-564c-4a0d-9b14-4bce14a4f070-kubelet-config\") pod \"global-pull-secret-syncer-pz2h5\" (UID: \"e089ab9b-564c-4a0d-9b14-4bce14a4f070\") " pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:33.082327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080241 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e089ab9b-564c-4a0d-9b14-4bce14a4f070-kubelet-config\") pod \"global-pull-secret-syncer-pz2h5\" (UID: \"e089ab9b-564c-4a0d-9b14-4bce14a4f070\") " pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:33.082327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080304 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-run-systemd\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.082327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080372 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fac37a7e-282e-4f8f-b1b1-1c259d141c58-sys\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.082327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080382 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:33.082327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080389 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-ovnkube-config\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.082327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080424 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6432496f-c87d-403d-bdaf-f54af3038e6a-device-dir\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:33.082327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080427 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-run-openvswitch\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.082891 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080513 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a323a5a4-1f3f-49b5-ae58-8b5c90592e5a-hosts-file\") pod \"node-resolver-lrfwx\" (UID: \"a323a5a4-1f3f-49b5-ae58-8b5c90592e5a\") " pod="openshift-dns/node-resolver-lrfwx" Jul 15 19:21:33.082891 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080816 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-env-overrides\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.082891 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080816 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/98333cd6-ba91-4b38-bf05-64e47d2f0251-serviceca\") pod \"node-ca-n6mjl\" (UID: \"98333cd6-ba91-4b38-bf05-64e47d2f0251\") " pod="openshift-image-registry/node-ca-n6mjl" Jul 15 19:21:33.082891 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.080862 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-cni-binary-copy\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:33.082891 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.081337 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fac37a7e-282e-4f8f-b1b1-1c259d141c58-tmp\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.082891 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.081912 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fac37a7e-282e-4f8f-b1b1-1c259d141c58-etc-tuned\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.082891 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.082419 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-ovn-node-metrics-cert\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.089310 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.089257 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcnwb\" (UniqueName: \"kubernetes.io/projected/a323a5a4-1f3f-49b5-ae58-8b5c90592e5a-kube-api-access-xcnwb\") pod \"node-resolver-lrfwx\" (UID: \"a323a5a4-1f3f-49b5-ae58-8b5c90592e5a\") " pod="openshift-dns/node-resolver-lrfwx" Jul 15 19:21:33.090818 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.090795 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwjqv\" (UniqueName: \"kubernetes.io/projected/8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d-kube-api-access-rwjqv\") pod \"ovnkube-node-rr9n5\" (UID: \"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.090966 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.090948 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwx4t\" (UniqueName: \"kubernetes.io/projected/6fb28f74-8890-418f-97f3-6da7ba5bc4a8-kube-api-access-wwx4t\") pod \"multus-additional-cni-plugins-5blhm\" (UID: \"6fb28f74-8890-418f-97f3-6da7ba5bc4a8\") " pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:33.091092 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.091060 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rqvz\" (UniqueName: \"kubernetes.io/projected/0a0dc43c-4fbe-4d98-aabe-831e75cda5ff-kube-api-access-2rqvz\") pod \"iptables-alerter-ddf8s\" (UID: \"0a0dc43c-4fbe-4d98-aabe-831e75cda5ff\") " pod="openshift-network-operator/iptables-alerter-ddf8s" Jul 15 19:21:33.091188 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.091065 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sbz5\" (UniqueName: \"kubernetes.io/projected/98333cd6-ba91-4b38-bf05-64e47d2f0251-kube-api-access-7sbz5\") pod \"node-ca-n6mjl\" (UID: \"98333cd6-ba91-4b38-bf05-64e47d2f0251\") " pod="openshift-image-registry/node-ca-n6mjl" Jul 15 19:21:33.091386 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.091369 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwhv2\" (UniqueName: \"kubernetes.io/projected/fac37a7e-282e-4f8f-b1b1-1c259d141c58-kube-api-access-lwhv2\") pod \"tuned-w24jl\" (UID: \"fac37a7e-282e-4f8f-b1b1-1c259d141c58\") " pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.091428 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.091377 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxkcn\" (UniqueName: \"kubernetes.io/projected/6432496f-c87d-403d-bdaf-f54af3038e6a-kube-api-access-fxkcn\") pod \"aws-ebs-csi-driver-node-flmhj\" (UID: \"6432496f-c87d-403d-bdaf-f54af3038e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:33.195286 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.195208 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mwkrg" Jul 15 19:21:33.204528 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:33.204503 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004b9afd_c0c0_48b7_8b25_f3199c2af4a2.slice/crio-dbcb74bddbda24207f2dc75a675b79506ed58b74420b2e3457fb9ec9bd00886d WatchSource:0}: Error finding container dbcb74bddbda24207f2dc75a675b79506ed58b74420b2e3457fb9ec9bd00886d: Status 404 returned error can't find the container with id dbcb74bddbda24207f2dc75a675b79506ed58b74420b2e3457fb9ec9bd00886d Jul 15 19:21:33.207869 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.207850 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g6pfk" Jul 15 19:21:33.213979 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.213961 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lrfwx" Jul 15 19:21:33.215333 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:33.215312 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a3fc5a_d407_4ad9_9862_e5fcda33d18e.slice/crio-0a7c8755f641e9b3239562bd6a34daa5c6a3784c2a2663e9786d6fd0de359321 WatchSource:0}: Error finding container 0a7c8755f641e9b3239562bd6a34daa5c6a3784c2a2663e9786d6fd0de359321: Status 404 returned error can't find the container with id 0a7c8755f641e9b3239562bd6a34daa5c6a3784c2a2663e9786d6fd0de359321 Jul 15 19:21:33.222578 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:33.222521 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda323a5a4_1f3f_49b5_ae58_8b5c90592e5a.slice/crio-845fe3ca8dec3b6faee87b4b43104ba45ec39c2b0e5470603980643e9874c8a1 WatchSource:0}: Error finding container 845fe3ca8dec3b6faee87b4b43104ba45ec39c2b0e5470603980643e9874c8a1: Status 404 returned error can't find the container with id 845fe3ca8dec3b6faee87b4b43104ba45ec39c2b0e5470603980643e9874c8a1 Jul 15 19:21:33.230741 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.230724 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w24jl" Jul 15 19:21:33.237605 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.237586 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:21:33.237838 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:33.237820 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac37a7e_282e_4f8f_b1b1_1c259d141c58.slice/crio-55d5e238b64b191a6e9a691c2e475a79ed90423a3ba173581fd828c46be72fae WatchSource:0}: Error finding container 55d5e238b64b191a6e9a691c2e475a79ed90423a3ba173581fd828c46be72fae: Status 404 returned error can't find the container with id 55d5e238b64b191a6e9a691c2e475a79ed90423a3ba173581fd828c46be72fae Jul 15 19:21:33.245150 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:33.245127 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1c7e79_6ca1_40b9_99a5_2fd7e55c8c4d.slice/crio-9f81ed3c2a9da5e04a74f0f769cb5ba9839caff9129bf24a522415f4e2538ed3 WatchSource:0}: Error finding container 9f81ed3c2a9da5e04a74f0f769cb5ba9839caff9129bf24a522415f4e2538ed3: Status 404 returned error can't find the container with id 9f81ed3c2a9da5e04a74f0f769cb5ba9839caff9129bf24a522415f4e2538ed3 Jul 15 19:21:33.274858 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.274828 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" Jul 15 19:21:33.282291 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:33.282265 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6432496f_c87d_403d_bdaf_f54af3038e6a.slice/crio-bc73282ff77573ab1ee83f49609b8151fd6d6424c4039eb3c42cd93c786d354c WatchSource:0}: Error finding container bc73282ff77573ab1ee83f49609b8151fd6d6424c4039eb3c42cd93c786d354c: Status 404 returned error can't find the container with id bc73282ff77573ab1ee83f49609b8151fd6d6424c4039eb3c42cd93c786d354c Jul 15 19:21:33.300817 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.300789 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n6mjl" Jul 15 19:21:33.308733 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:33.308712 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98333cd6_ba91_4b38_bf05_64e47d2f0251.slice/crio-d7270fe870a294303ede839f9653f6406f61a6908c77208558f56bf4a4748481 WatchSource:0}: Error finding container d7270fe870a294303ede839f9653f6406f61a6908c77208558f56bf4a4748481: Status 404 returned error can't find the container with id d7270fe870a294303ede839f9653f6406f61a6908c77208558f56bf4a4748481 Jul 15 19:21:33.315003 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.314977 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5blhm" Jul 15 19:21:33.320470 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.320447 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ddf8s" Jul 15 19:21:33.323773 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:33.323750 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb28f74_8890_418f_97f3_6da7ba5bc4a8.slice/crio-ddf88a19cf8d1f38cfcf8964d0c5e14407c269d2b08f9adbc9a3fff0cd424aee WatchSource:0}: Error finding container ddf88a19cf8d1f38cfcf8964d0c5e14407c269d2b08f9adbc9a3fff0cd424aee: Status 404 returned error can't find the container with id ddf88a19cf8d1f38cfcf8964d0c5e14407c269d2b08f9adbc9a3fff0cd424aee Jul 15 19:21:33.332580 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:21:33.332540 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0dc43c_4fbe_4d98_aabe_831e75cda5ff.slice/crio-b7d5733516e11be062ed8d28c51b7fe5f25b490301bf896b50af87e8cbd43d18 WatchSource:0}: Error finding container b7d5733516e11be062ed8d28c51b7fe5f25b490301bf896b50af87e8cbd43d18: Status 404 returned error can't find the container with id b7d5733516e11be062ed8d28c51b7fe5f25b490301bf896b50af87e8cbd43d18 Jul 15 19:21:33.483547 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.483444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kdzv\" (UniqueName: \"kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv\") pod \"network-check-target-5w4r4\" (UID: \"5d03ee98-6e6f-4ee1-8b78-d4da30c503ff\") " pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:33.483547 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.483510 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs\") pod \"network-metrics-daemon-zfvrb\" (UID: \"1a59a892-2494-489b-b5f2-038c7161cf7e\") " pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:33.483764 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:33.483633 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:21:33.483764 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:33.483641 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:21:33.483764 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:33.483671 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:21:33.483764 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:33.483686 2564 projected.go:196] Error preparing data for projected volume kube-api-access-5kdzv for pod openshift-network-diagnostics/network-check-target-5w4r4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:21:33.483764 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:33.483704 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs podName:1a59a892-2494-489b-b5f2-038c7161cf7e nodeName:}" failed. No retries permitted until 2026-07-15 19:21:34.483684338 +0000 UTC m=+2.955159609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs") pod "network-metrics-daemon-zfvrb" (UID: "1a59a892-2494-489b-b5f2-038c7161cf7e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:21:33.483764 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:33.483740 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv podName:5d03ee98-6e6f-4ee1-8b78-d4da30c503ff nodeName:}" failed. No retries permitted until 2026-07-15 19:21:34.483721862 +0000 UTC m=+2.955197148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5kdzv" (UniqueName: "kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv") pod "network-check-target-5w4r4" (UID: "5d03ee98-6e6f-4ee1-8b78-d4da30c503ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:21:33.585136 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.584611 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret\") pod \"global-pull-secret-syncer-pz2h5\" (UID: \"e089ab9b-564c-4a0d-9b14-4bce14a4f070\") " pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:33.585136 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:33.584768 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:21:33.585136 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:33.584825 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret podName:e089ab9b-564c-4a0d-9b14-4bce14a4f070 nodeName:}" failed. No retries permitted until 2026-07-15 19:21:34.584810696 +0000 UTC m=+3.056285965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret") pod "global-pull-secret-syncer-pz2h5" (UID: "e089ab9b-564c-4a0d-9b14-4bce14a4f070") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:21:33.697828 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.697796 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:21:33.975884 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.975763 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 19:16:33 +0000 UTC" deadline="2028-01-14 20:30:37.498552586 +0000 UTC" Jul 15 19:21:33.975884 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:33.975830 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13153h9m3.522726991s" Jul 15 19:21:34.030525 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:34.030494 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:21:34.078607 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:34.078519 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5blhm" event={"ID":"6fb28f74-8890-418f-97f3-6da7ba5bc4a8","Type":"ContainerStarted","Data":"ddf88a19cf8d1f38cfcf8964d0c5e14407c269d2b08f9adbc9a3fff0cd424aee"} Jul 15 19:21:34.090257 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:34.090217 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n6mjl" event={"ID":"98333cd6-ba91-4b38-bf05-64e47d2f0251","Type":"ContainerStarted","Data":"d7270fe870a294303ede839f9653f6406f61a6908c77208558f56bf4a4748481"} Jul 15 19:21:34.096054 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:34.096023 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" event={"ID":"6432496f-c87d-403d-bdaf-f54af3038e6a","Type":"ContainerStarted","Data":"bc73282ff77573ab1ee83f49609b8151fd6d6424c4039eb3c42cd93c786d354c"} Jul 15 19:21:34.110134 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:34.110106 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" event={"ID":"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d","Type":"ContainerStarted","Data":"9f81ed3c2a9da5e04a74f0f769cb5ba9839caff9129bf24a522415f4e2538ed3"} Jul 15 19:21:34.119689 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:34.119661 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g6pfk" event={"ID":"d2a3fc5a-d407-4ad9-9862-e5fcda33d18e","Type":"ContainerStarted","Data":"0a7c8755f641e9b3239562bd6a34daa5c6a3784c2a2663e9786d6fd0de359321"} Jul 15 19:21:34.129809 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:34.129781 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mwkrg" event={"ID":"004b9afd-c0c0-48b7-8b25-f3199c2af4a2","Type":"ContainerStarted","Data":"dbcb74bddbda24207f2dc75a675b79506ed58b74420b2e3457fb9ec9bd00886d"} Jul 15 19:21:34.150310 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:34.150281 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-225.ec2.internal" event={"ID":"90a23a491012cfceb92edd519600eade","Type":"ContainerStarted","Data":"83630cceddcb70cdbdc998fdda73a3e5555643fa5ed6b818b281eed6ecc3b87c"} Jul 15 19:21:34.152904 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:34.152839 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal" event={"ID":"e3083601b4042ee5819adc3570b16f90","Type":"ContainerStarted","Data":"5c7eef9260259815c12ea6673b17b46cf63fe69b4fa9c9170e8d802652bb25f1"} Jul 15 19:21:34.155923 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:34.155894 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ddf8s" event={"ID":"0a0dc43c-4fbe-4d98-aabe-831e75cda5ff","Type":"ContainerStarted","Data":"b7d5733516e11be062ed8d28c51b7fe5f25b490301bf896b50af87e8cbd43d18"} Jul 15 19:21:34.169453 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:34.169424 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w24jl" event={"ID":"fac37a7e-282e-4f8f-b1b1-1c259d141c58","Type":"ContainerStarted","Data":"55d5e238b64b191a6e9a691c2e475a79ed90423a3ba173581fd828c46be72fae"} Jul 15 19:21:34.179357 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:34.179327 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lrfwx" event={"ID":"a323a5a4-1f3f-49b5-ae58-8b5c90592e5a","Type":"ContainerStarted","Data":"845fe3ca8dec3b6faee87b4b43104ba45ec39c2b0e5470603980643e9874c8a1"} Jul 15 19:21:34.497107 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:34.496841 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kdzv\" (UniqueName: \"kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv\") pod \"network-check-target-5w4r4\" (UID: \"5d03ee98-6e6f-4ee1-8b78-d4da30c503ff\") " pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:34.497291 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:34.497143 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs\") pod \"network-metrics-daemon-zfvrb\" (UID: \"1a59a892-2494-489b-b5f2-038c7161cf7e\") " pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:34.497435 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:34.497414 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:21:34.497545 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:34.497489 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs podName:1a59a892-2494-489b-b5f2-038c7161cf7e nodeName:}" failed. No retries permitted until 2026-07-15 19:21:36.497468329 +0000 UTC m=+4.968943622 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs") pod "network-metrics-daemon-zfvrb" (UID: "1a59a892-2494-489b-b5f2-038c7161cf7e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:21:34.497729 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:34.497703 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:21:34.497809 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:34.497742 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:21:34.497809 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:34.497770 2564 projected.go:196] Error preparing data for projected volume kube-api-access-5kdzv for pod openshift-network-diagnostics/network-check-target-5w4r4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:21:34.497919 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:34.497844 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv podName:5d03ee98-6e6f-4ee1-8b78-d4da30c503ff nodeName:}" failed. No retries permitted until 2026-07-15 19:21:36.497812983 +0000 UTC m=+4.969288256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5kdzv" (UniqueName: "kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv") pod "network-check-target-5w4r4" (UID: "5d03ee98-6e6f-4ee1-8b78-d4da30c503ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:21:34.598100 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:34.597817 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret\") pod \"global-pull-secret-syncer-pz2h5\" (UID: \"e089ab9b-564c-4a0d-9b14-4bce14a4f070\") " pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:34.598100 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:34.597960 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:21:34.598100 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:34.598022 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret podName:e089ab9b-564c-4a0d-9b14-4bce14a4f070 nodeName:}" failed. No retries permitted until 2026-07-15 19:21:36.598001825 +0000 UTC m=+5.069477097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret") pod "global-pull-secret-syncer-pz2h5" (UID: "e089ab9b-564c-4a0d-9b14-4bce14a4f070") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:21:34.976028 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:34.975948 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 19:16:33 +0000 UTC" deadline="2028-04-14 12:34:16.562429046 +0000 UTC" Jul 15 19:21:34.976028 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:34.975993 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15329h12m41.586440998s" Jul 15 19:21:35.053888 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:35.053849 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:35.054083 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:35.053991 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:21:35.054419 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:35.054391 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:35.054532 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:35.054465 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:21:35.054532 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:35.054513 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:35.054647 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:35.054582 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:21:35.998742 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:35.998710 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:21:36.513891 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:36.513859 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kdzv\" (UniqueName: \"kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv\") pod \"network-check-target-5w4r4\" (UID: \"5d03ee98-6e6f-4ee1-8b78-d4da30c503ff\") " pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:36.514090 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:36.513915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs\") pod \"network-metrics-daemon-zfvrb\" (UID: \"1a59a892-2494-489b-b5f2-038c7161cf7e\") " pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:36.514090 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:36.514050 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:21:36.514216 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:36.514114 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs podName:1a59a892-2494-489b-b5f2-038c7161cf7e nodeName:}" failed. No retries permitted until 2026-07-15 19:21:40.514095382 +0000 UTC m=+8.985570656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs") pod "network-metrics-daemon-zfvrb" (UID: "1a59a892-2494-489b-b5f2-038c7161cf7e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:21:36.514774 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:36.514461 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:21:36.514774 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:36.514494 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:21:36.514774 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:36.514508 2564 projected.go:196] Error preparing data for projected volume kube-api-access-5kdzv for pod openshift-network-diagnostics/network-check-target-5w4r4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:21:36.514774 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:36.514587 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv podName:5d03ee98-6e6f-4ee1-8b78-d4da30c503ff nodeName:}" failed. No retries permitted until 2026-07-15 19:21:40.51456866 +0000 UTC m=+8.986043935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5kdzv" (UniqueName: "kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv") pod "network-check-target-5w4r4" (UID: "5d03ee98-6e6f-4ee1-8b78-d4da30c503ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:21:36.615722 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:36.614916 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret\") pod \"global-pull-secret-syncer-pz2h5\" (UID: \"e089ab9b-564c-4a0d-9b14-4bce14a4f070\") " pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:36.615722 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:36.615151 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:21:36.615722 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:36.615221 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret podName:e089ab9b-564c-4a0d-9b14-4bce14a4f070 nodeName:}" failed. No retries permitted until 2026-07-15 19:21:40.615200342 +0000 UTC m=+9.086675611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret") pod "global-pull-secret-syncer-pz2h5" (UID: "e089ab9b-564c-4a0d-9b14-4bce14a4f070") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:21:37.053845 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:37.053793 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:37.054335 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:37.053956 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:21:37.054423 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:37.054394 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:37.054567 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:37.054497 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:21:37.054656 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:37.054573 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:37.054707 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:37.054646 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:21:39.053849 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:39.053809 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:39.054260 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:39.053951 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:21:39.054260 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:39.054035 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:39.054260 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:39.054107 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:21:39.054260 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:39.054161 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:39.054260 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:39.054247 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:21:40.554223 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:40.554178 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kdzv\" (UniqueName: \"kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv\") pod \"network-check-target-5w4r4\" (UID: \"5d03ee98-6e6f-4ee1-8b78-d4da30c503ff\") " pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:40.554629 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:40.554245 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs\") pod \"network-metrics-daemon-zfvrb\" (UID: \"1a59a892-2494-489b-b5f2-038c7161cf7e\") " pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:40.554629 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:40.554432 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:21:40.554629 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:40.554499 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs podName:1a59a892-2494-489b-b5f2-038c7161cf7e nodeName:}" failed. No retries permitted until 2026-07-15 19:21:48.554478339 +0000 UTC m=+17.025953612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs") pod "network-metrics-daemon-zfvrb" (UID: "1a59a892-2494-489b-b5f2-038c7161cf7e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:21:40.554629 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:40.554617 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:21:40.554843 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:40.554638 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:21:40.554843 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:40.554652 2564 projected.go:196] Error preparing data for projected volume kube-api-access-5kdzv for pod openshift-network-diagnostics/network-check-target-5w4r4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:21:40.554843 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:40.554693 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv podName:5d03ee98-6e6f-4ee1-8b78-d4da30c503ff nodeName:}" failed. No retries permitted until 2026-07-15 19:21:48.554679816 +0000 UTC m=+17.026155099 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5kdzv" (UniqueName: "kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv") pod "network-check-target-5w4r4" (UID: "5d03ee98-6e6f-4ee1-8b78-d4da30c503ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:21:40.655271 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:40.654735 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret\") pod \"global-pull-secret-syncer-pz2h5\" (UID: \"e089ab9b-564c-4a0d-9b14-4bce14a4f070\") " pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:40.655271 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:40.654900 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:21:40.655271 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:40.654967 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret podName:e089ab9b-564c-4a0d-9b14-4bce14a4f070 nodeName:}" failed. No retries permitted until 2026-07-15 19:21:48.654946426 +0000 UTC m=+17.126421701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret") pod "global-pull-secret-syncer-pz2h5" (UID: "e089ab9b-564c-4a0d-9b14-4bce14a4f070") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:21:41.053707 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:41.053622 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:41.053861 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:41.053723 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:41.053861 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:41.053837 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:21:41.054239 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:41.054219 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:41.054346 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:41.054325 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:21:41.054450 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:41.054432 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:21:43.054844 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:43.053957 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:43.054844 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:43.054103 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:21:43.054844 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:43.054520 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:43.054844 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:43.054631 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:21:43.054844 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:43.054704 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:43.054844 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:43.054796 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:21:45.053784 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:45.053745 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:45.054218 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:45.053747 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:45.054218 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:45.053863 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:21:45.054218 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:45.053981 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:21:45.054218 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:45.053757 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:45.054218 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:45.054089 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:21:47.054355 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:47.054324 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:47.054355 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:47.054356 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:47.054880 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:47.054325 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:47.054880 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:47.054490 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:21:47.054880 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:47.054545 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:21:47.054880 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:47.054665 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:21:48.625164 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:48.625121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kdzv\" (UniqueName: \"kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv\") pod \"network-check-target-5w4r4\" (UID: \"5d03ee98-6e6f-4ee1-8b78-d4da30c503ff\") " pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:48.625164 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:48.625173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs\") pod \"network-metrics-daemon-zfvrb\" (UID: \"1a59a892-2494-489b-b5f2-038c7161cf7e\") " pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:48.625738 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:48.625552 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:21:48.625738 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:48.625595 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:21:48.625738 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:48.625611 2564 projected.go:196] Error preparing data for projected volume kube-api-access-5kdzv for pod openshift-network-diagnostics/network-check-target-5w4r4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:21:48.625738 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:48.625682 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv podName:5d03ee98-6e6f-4ee1-8b78-d4da30c503ff nodeName:}" failed. No retries permitted until 2026-07-15 19:22:04.625663076 +0000 UTC m=+33.097138362 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5kdzv" (UniqueName: "kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv") pod "network-check-target-5w4r4" (UID: "5d03ee98-6e6f-4ee1-8b78-d4da30c503ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:21:48.625738 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:48.625695 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:21:48.625892 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:48.625764 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs podName:1a59a892-2494-489b-b5f2-038c7161cf7e nodeName:}" failed. No retries permitted until 2026-07-15 19:22:04.625750534 +0000 UTC m=+33.097225802 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs") pod "network-metrics-daemon-zfvrb" (UID: "1a59a892-2494-489b-b5f2-038c7161cf7e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:21:48.726122 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:48.726084 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret\") pod \"global-pull-secret-syncer-pz2h5\" (UID: \"e089ab9b-564c-4a0d-9b14-4bce14a4f070\") " pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:48.726268 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:48.726212 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:21:48.726325 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:48.726296 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret podName:e089ab9b-564c-4a0d-9b14-4bce14a4f070 nodeName:}" failed. No retries permitted until 2026-07-15 19:22:04.726280172 +0000 UTC m=+33.197755438 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret") pod "global-pull-secret-syncer-pz2h5" (UID: "e089ab9b-564c-4a0d-9b14-4bce14a4f070") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:21:49.054542 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:49.054440 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:49.054693 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:49.054442 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:49.054693 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:49.054597 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:21:49.054693 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:49.054615 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:49.054798 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:49.054704 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:21:49.054798 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:49.054759 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:21:51.054419 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:51.054337 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:51.054419 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:51.054384 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:51.054955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:51.054458 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:51.054955 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:51.054485 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:21:51.054955 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:51.054560 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:21:51.054955 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:51.054666 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:21:53.054010 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:53.053976 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:53.054434 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:53.053976 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:53.054434 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:53.054118 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:21:53.054434 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:53.053975 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:53.054434 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:53.054196 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:21:53.054434 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:53.054253 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:21:55.054570 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.054400 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:55.055164 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.054448 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:55.055164 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:55.054665 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:21:55.055164 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.054454 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:55.055164 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:55.054750 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:21:55.055164 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:55.054878 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:21:55.234355 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.234178 2564 generic.go:356] "Generic (PLEG): container finished" podID="e3083601b4042ee5819adc3570b16f90" containerID="7a13b9e5b227529411ed464e2b62308bee57d421d880f526cbcc2d710c333271" exitCode=0 Jul 15 19:21:55.234451 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.234221 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal" event={"ID":"e3083601b4042ee5819adc3570b16f90","Type":"ContainerDied","Data":"7a13b9e5b227529411ed464e2b62308bee57d421d880f526cbcc2d710c333271"} Jul 15 19:21:55.236087 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.236033 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w24jl" event={"ID":"fac37a7e-282e-4f8f-b1b1-1c259d141c58","Type":"ContainerStarted","Data":"aeaf14a3d450b98ce1ca09059a1e3d792236b1e79b0032cccdff9816dcf242f6"} Jul 15 19:21:55.237329 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.237308 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lrfwx" event={"ID":"a323a5a4-1f3f-49b5-ae58-8b5c90592e5a","Type":"ContainerStarted","Data":"de0f5c1624fbe14327bb8869a9cb50e511c9d6936c28e09da36bdc0370d6b42b"} Jul 15 19:21:55.238895 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.238873 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5blhm" event={"ID":"6fb28f74-8890-418f-97f3-6da7ba5bc4a8","Type":"ContainerStarted","Data":"f752e4f9f5429a3b6e0fb7cfebb11bb3b919c8e8138fa81073524c35ddf93c61"} Jul 15 19:21:55.240200 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.240182 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n6mjl" event={"ID":"98333cd6-ba91-4b38-bf05-64e47d2f0251","Type":"ContainerStarted","Data":"835b7eae0af90cf4c3793b2912e6534ec8b7484d106541da167a13e9a970d3e1"} Jul 15 19:21:55.241517 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.241496 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" event={"ID":"6432496f-c87d-403d-bdaf-f54af3038e6a","Type":"ContainerStarted","Data":"a47b209d25775382a112b91cc1679bd87445b3d9fcc72920d426af4097070fe0"} Jul 15 19:21:55.243794 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.243770 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" event={"ID":"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d","Type":"ContainerStarted","Data":"ea530d0b06886a77aa3a196655e41bc6a6ea846fe35f05cd76e5d9897139bb94"} Jul 15 19:21:55.243888 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.243839 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" event={"ID":"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d","Type":"ContainerStarted","Data":"b428dbbf593d2b7603761e28844517e3d2ab2eba41aef5a8f7bc8edbbf1057f7"} Jul 15 19:21:55.243948 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.243905 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" event={"ID":"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d","Type":"ContainerStarted","Data":"00ec7bbe617a46a9f73356e9dead7ade6dfa906f27cc77104cf1221268dd6ef3"} Jul 15 19:21:55.243987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.243951 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" event={"ID":"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d","Type":"ContainerStarted","Data":"82166fbaf34ab78facf046507882c5a56c89764ec66ba81040a79fd21a271d16"} Jul 15 19:21:55.245139 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.245120 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g6pfk" event={"ID":"d2a3fc5a-d407-4ad9-9862-e5fcda33d18e","Type":"ContainerStarted","Data":"d134182ed28c6ff7cbddf35e242de6270b54a2f47ba464335545adecca114a38"} Jul 15 19:21:55.247530 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.247086 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mwkrg" event={"ID":"004b9afd-c0c0-48b7-8b25-f3199c2af4a2","Type":"ContainerStarted","Data":"24ec80b18c701eb78790f94964e79fc840c8707a97934bdc331410efe28d757a"} Jul 15 19:21:55.249270 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.249246 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-225.ec2.internal" event={"ID":"90a23a491012cfceb92edd519600eade","Type":"ContainerStarted","Data":"d65d935ec7d92dd41da523f7896651ef6dd3cfddb97e2db3a13d5bcb6f1aef31"} Jul 15 19:21:55.275268 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.275228 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-225.ec2.internal" podStartSLOduration=22.275214245 podStartE2EDuration="22.275214245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:21:55.274985933 +0000 UTC m=+23.746461223" watchObservedRunningTime="2026-07-15 19:21:55.275214245 +0000 UTC m=+23.746689534" Jul 15 19:21:55.306717 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.306659 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mwkrg" podStartSLOduration=1.761951904 podStartE2EDuration="23.306625714s" totalImagesPullingTime="21.54467381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:21:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:21:55.292408239 +0000 UTC m=+23.763883529" watchObservedRunningTime="2026-07-15 19:21:55.306625714 +0000 UTC m=+23.778101010" Jul 15 19:21:55.324228 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.324177 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w24jl" podStartSLOduration=1.089388194 podStartE2EDuration="22.324156467s" totalImagesPullingTime="21.234768273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:21:55.323955602 +0000 UTC m=+23.795430891" watchObservedRunningTime="2026-07-15 19:21:55.324156467 +0000 UTC m=+23.795631755" Jul 15 19:21:55.324359 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.324331 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lrfwx" podStartSLOduration=1.105195329 podStartE2EDuration="22.324321248s" totalImagesPullingTime="21.219125919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:21:55.30705138 +0000 UTC m=+23.778526669" watchObservedRunningTime="2026-07-15 19:21:55.324321248 +0000 UTC m=+23.795796540" Jul 15 19:21:55.339805 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.339768 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n6mjl" podStartSLOduration=1.206840603 podStartE2EDuration="22.339753813s" totalImagesPullingTime="21.13291321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:21:55.33929547 +0000 UTC m=+23.810770762" watchObservedRunningTime="2026-07-15 19:21:55.339753813 +0000 UTC m=+23.811229102" Jul 15 19:21:55.362743 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.362699 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g6pfk" podStartSLOduration=2.136478229 podStartE2EDuration="23.362679454s" totalImagesPullingTime="21.226201225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:21:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:21:55.36267554 +0000 UTC m=+23.834150806" watchObservedRunningTime="2026-07-15 19:21:55.362679454 +0000 UTC m=+23.834154746" Jul 15 19:21:55.847622 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.847539 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g6pfk" Jul 15 19:21:55.848127 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:55.848112 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g6pfk" Jul 15 19:21:56.253318 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:56.253234 2564 generic.go:356] "Generic (PLEG): container finished" podID="6fb28f74-8890-418f-97f3-6da7ba5bc4a8" containerID="f752e4f9f5429a3b6e0fb7cfebb11bb3b919c8e8138fa81073524c35ddf93c61" exitCode=0 Jul 15 19:21:56.253768 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:56.253320 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5blhm" event={"ID":"6fb28f74-8890-418f-97f3-6da7ba5bc4a8","Type":"ContainerDied","Data":"f752e4f9f5429a3b6e0fb7cfebb11bb3b919c8e8138fa81073524c35ddf93c61"} Jul 15 19:21:56.256409 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:56.256383 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" event={"ID":"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d","Type":"ContainerStarted","Data":"f4f3e91ae047803d4dacc70081e1a9ea591f6876451546b565eca3ffcc02e23f"} Jul 15 19:21:56.256532 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:56.256485 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" event={"ID":"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d","Type":"ContainerStarted","Data":"044f68d673899f0a18aa3f503dc896fb5887a80752e6a8d8bd6701faabeb1b16"} Jul 15 19:21:56.258119 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:56.258100 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal" event={"ID":"e3083601b4042ee5819adc3570b16f90","Type":"ContainerStarted","Data":"baf6afd166dd92a81f434dc58c8472c227de779e5798838e5803a167370c6532"} Jul 15 19:21:56.259590 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:56.259566 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ddf8s" event={"ID":"0a0dc43c-4fbe-4d98-aabe-831e75cda5ff","Type":"ContainerStarted","Data":"647876904f3397e683063cce19b4afe5fcf7b233a44df0c25f5fc193f3124669"} Jul 15 19:21:56.259838 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:56.259816 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g6pfk" Jul 15 19:21:56.260283 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:56.260264 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g6pfk" Jul 15 19:21:56.287148 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:56.287098 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-225.ec2.internal" podStartSLOduration=23.287084557 podStartE2EDuration="23.287084557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:21:56.286787281 +0000 UTC m=+24.758262570" watchObservedRunningTime="2026-07-15 19:21:56.287084557 +0000 UTC m=+24.758559845" Jul 15 19:21:56.332233 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:56.332184 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ddf8s" podStartSLOduration=5.983088499 podStartE2EDuration="23.332169361s" totalImagesPullingTime="17.349080862s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:21:56.331805495 +0000 UTC m=+24.803280783" watchObservedRunningTime="2026-07-15 19:21:56.332169361 +0000 UTC m=+24.803644649" Jul 15 19:21:56.972759 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:56.972724 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 19:21:57.054044 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:57.054009 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:57.054228 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:57.054066 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:57.054228 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:57.054176 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:57.054228 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:57.054192 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:21:57.054401 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:57.054251 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:21:57.054401 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:57.054358 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:21:57.264667 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:57.264585 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" event={"ID":"6432496f-c87d-403d-bdaf-f54af3038e6a","Type":"ContainerStarted","Data":"d4a28284ea163034a1b6d5cd05ebe028e21a1d226c5217d027104faacc091510"} Jul 15 19:21:57.939067 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:57.938900 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T19:21:56.972752443Z","UUID":"19b5e3a0-b6ac-4404-ab8b-352e9869958c","Handler":null,"Name":"","Endpoint":""} Jul 15 19:21:57.940415 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:57.940396 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 19:21:57.940561 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:57.940432 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 19:21:58.271282 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:58.271251 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" event={"ID":"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d","Type":"ContainerStarted","Data":"97261c1691470e199ec2d9029a4a62b13330b87433bdff50b6234e82cfa22392"} Jul 15 19:21:59.053934 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:59.053710 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:21:59.054112 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:59.053762 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:21:59.054112 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:59.054014 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:21:59.054112 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:59.053762 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:21:59.054238 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:59.054109 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:21:59.054238 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:21:59.054163 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:21:59.275987 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:59.275938 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" event={"ID":"6432496f-c87d-403d-bdaf-f54af3038e6a","Type":"ContainerStarted","Data":"e154c868ed32b11ee3121d9bf4eebbab6117efcf656dc47d192af3d05b6697fa"} Jul 15 19:21:59.296019 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:21:59.295955 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flmhj" podStartSLOduration=2.255278605 podStartE2EDuration="26.295934848s" totalImagesPullingTime="24.040656243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:21:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:21:59.295567578 +0000 UTC m=+27.767042867" watchObservedRunningTime="2026-07-15 19:21:59.295934848 +0000 UTC m=+27.767410140" Jul 15 19:22:01.054653 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:01.054442 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:22:01.055246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:01.054500 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:22:01.055246 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:01.054742 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:22:01.055246 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:01.054546 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:22:01.055246 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:01.054846 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:22:01.055246 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:01.054942 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:22:01.281356 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:01.281273 2564 generic.go:356] "Generic (PLEG): container finished" podID="6fb28f74-8890-418f-97f3-6da7ba5bc4a8" containerID="57bd06fca35d68267a8164e819dcbc2903932827c448107e4b392736854d3d19" exitCode=0 Jul 15 19:22:01.281596 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:01.281365 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5blhm" event={"ID":"6fb28f74-8890-418f-97f3-6da7ba5bc4a8","Type":"ContainerDied","Data":"57bd06fca35d68267a8164e819dcbc2903932827c448107e4b392736854d3d19"} Jul 15 19:22:01.286642 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:01.286607 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" event={"ID":"8b1c7e79-6ca1-40b9-99a5-2fd7e55c8c4d","Type":"ContainerStarted","Data":"649b1a427105d977418cd9b128874f7621f4f1d03ccc1f3a61b34a8747834fbb"} Jul 15 19:22:01.286969 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:01.286934 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:22:01.302676 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:01.302652 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:22:01.326120 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:01.326062 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" podStartSLOduration=6.844627612 podStartE2EDuration="28.326047076s" totalImagesPullingTime="21.481419464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:22:01.325577129 +0000 UTC m=+29.797052418" watchObservedRunningTime="2026-07-15 19:22:01.326047076 +0000 UTC m=+29.797522364" Jul 15 19:22:02.288780 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:02.288748 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:22:02.289313 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:02.289128 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:22:02.305077 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:02.305051 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:22:02.681149 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:02.681061 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zfvrb"] Jul 15 19:22:02.681315 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:02.681193 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:22:02.681379 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:02.681316 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:22:02.684418 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:02.684380 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pz2h5"] Jul 15 19:22:02.684584 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:02.684516 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:22:02.684634 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:02.684616 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:22:02.684974 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:02.684956 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5w4r4"] Jul 15 19:22:02.685050 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:02.685041 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:22:02.685135 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:02.685115 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:22:03.293141 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:03.293105 2564 generic.go:356] "Generic (PLEG): container finished" podID="6fb28f74-8890-418f-97f3-6da7ba5bc4a8" containerID="e0db97e84da3e75721682f50ede77532632f90ff7a4eb3809d7f76cc6d976965" exitCode=0 Jul 15 19:22:03.293525 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:03.293186 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5blhm" event={"ID":"6fb28f74-8890-418f-97f3-6da7ba5bc4a8","Type":"ContainerDied","Data":"e0db97e84da3e75721682f50ede77532632f90ff7a4eb3809d7f76cc6d976965"} Jul 15 19:22:04.056607 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:04.056579 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:22:04.056607 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:04.056592 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:22:04.056607 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:04.056593 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:22:04.056882 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:04.056668 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:22:04.056882 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:04.056785 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:22:04.056970 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:04.056894 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:22:04.637206 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:04.636934 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kdzv\" (UniqueName: \"kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv\") pod \"network-check-target-5w4r4\" (UID: \"5d03ee98-6e6f-4ee1-8b78-d4da30c503ff\") " pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:22:04.637206 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:04.637201 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs\") pod \"network-metrics-daemon-zfvrb\" (UID: \"1a59a892-2494-489b-b5f2-038c7161cf7e\") " pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:22:04.637625 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:04.637102 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:22:04.637625 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:04.637252 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:22:04.637625 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:04.637269 2564 projected.go:196] Error preparing data for projected volume kube-api-access-5kdzv for pod openshift-network-diagnostics/network-check-target-5w4r4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:22:04.637625 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:04.637325 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv podName:5d03ee98-6e6f-4ee1-8b78-d4da30c503ff nodeName:}" failed. No retries permitted until 2026-07-15 19:22:36.637308372 +0000 UTC m=+65.108783641 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5kdzv" (UniqueName: "kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv") pod "network-check-target-5w4r4" (UID: "5d03ee98-6e6f-4ee1-8b78-d4da30c503ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:22:04.637625 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:04.637333 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:22:04.637625 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:04.637368 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs podName:1a59a892-2494-489b-b5f2-038c7161cf7e nodeName:}" failed. No retries permitted until 2026-07-15 19:22:36.637356502 +0000 UTC m=+65.108831768 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs") pod "network-metrics-daemon-zfvrb" (UID: "1a59a892-2494-489b-b5f2-038c7161cf7e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:22:04.737918 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:04.737885 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret\") pod \"global-pull-secret-syncer-pz2h5\" (UID: \"e089ab9b-564c-4a0d-9b14-4bce14a4f070\") " pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:22:04.738086 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:04.738016 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:22:04.738086 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:04.738081 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret podName:e089ab9b-564c-4a0d-9b14-4bce14a4f070 nodeName:}" failed. No retries permitted until 2026-07-15 19:22:36.738062989 +0000 UTC m=+65.209538269 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret") pod "global-pull-secret-syncer-pz2h5" (UID: "e089ab9b-564c-4a0d-9b14-4bce14a4f070") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:22:05.300038 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:05.300001 2564 generic.go:356] "Generic (PLEG): container finished" podID="6fb28f74-8890-418f-97f3-6da7ba5bc4a8" containerID="921ed76c39987dcd5bd4968f82e78bd51f8e88bb24002910977337da46ae1046" exitCode=0 Jul 15 19:22:05.300232 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:05.300060 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5blhm" event={"ID":"6fb28f74-8890-418f-97f3-6da7ba5bc4a8","Type":"ContainerDied","Data":"921ed76c39987dcd5bd4968f82e78bd51f8e88bb24002910977337da46ae1046"} Jul 15 19:22:05.465699 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:05.465669 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:05.559216 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:05.559128 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:06.057849 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:06.057775 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:22:06.058273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:06.057775 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:22:06.058273 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:06.057900 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:22:06.058273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:06.057776 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:22:06.058273 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:06.057954 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pz2h5" podUID="e089ab9b-564c-4a0d-9b14-4bce14a4f070" Jul 15 19:22:06.058273 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:06.058021 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5w4r4" podUID="5d03ee98-6e6f-4ee1-8b78-d4da30c503ff" Jul 15 19:22:06.436605 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:06.436518 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:06.497568 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:06.497536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:07.357516 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.357480 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-225.ec2.internal" event="NodeReady" Jul 15 19:22:07.357938 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.357648 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 19:22:07.403412 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.403377 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xtdvl"] Jul 15 19:22:07.405759 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.405736 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:07.412092 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.412070 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bw2gg\"" Jul 15 19:22:07.412274 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.412259 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 19:22:07.412452 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.412435 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 19:22:07.412636 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.412618 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-p5kfg"] Jul 15 19:22:07.414550 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.414529 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p5kfg" Jul 15 19:22:07.416597 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.416579 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 19:22:07.416746 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.416579 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 19:22:07.416746 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.416735 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xtdvl"] Jul 15 19:22:07.416888 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.416654 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 19:22:07.417199 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.417049 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dghvh\"" Jul 15 19:22:07.421723 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.421668 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p5kfg"] Jul 15 19:22:07.553753 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.553717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2kgg\" (UniqueName: \"kubernetes.io/projected/26ff859f-ec36-43d8-888d-bba78cf3dffb-kube-api-access-f2kgg\") pod \"ingress-canary-p5kfg\" (UID: \"26ff859f-ec36-43d8-888d-bba78cf3dffb\") " pod="openshift-ingress-canary/ingress-canary-p5kfg" Jul 15 19:22:07.553929 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.553773 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ff859f-ec36-43d8-888d-bba78cf3dffb-cert\") pod \"ingress-canary-p5kfg\" (UID: \"26ff859f-ec36-43d8-888d-bba78cf3dffb\") " pod="openshift-ingress-canary/ingress-canary-p5kfg" Jul 15 19:22:07.553929 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.553868 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c110945-616b-4146-ac13-c20e4c84ef1e-tmp-dir\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:07.553929 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.553907 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:07.554074 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.553941 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9clf\" (UniqueName: \"kubernetes.io/projected/6c110945-616b-4146-ac13-c20e4c84ef1e-kube-api-access-r9clf\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:07.554074 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.553982 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c110945-616b-4146-ac13-c20e4c84ef1e-config-volume\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:07.655038 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.654997 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c110945-616b-4146-ac13-c20e4c84ef1e-tmp-dir\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:07.655217 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.655057 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:07.655217 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.655101 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9clf\" (UniqueName: \"kubernetes.io/projected/6c110945-616b-4146-ac13-c20e4c84ef1e-kube-api-access-r9clf\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:07.655217 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.655127 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c110945-616b-4146-ac13-c20e4c84ef1e-config-volume\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:07.655217 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:07.655156 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:22:07.655217 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.655171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2kgg\" (UniqueName: \"kubernetes.io/projected/26ff859f-ec36-43d8-888d-bba78cf3dffb-kube-api-access-f2kgg\") pod \"ingress-canary-p5kfg\" (UID: \"26ff859f-ec36-43d8-888d-bba78cf3dffb\") " pod="openshift-ingress-canary/ingress-canary-p5kfg" Jul 15 19:22:07.655217 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.655206 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ff859f-ec36-43d8-888d-bba78cf3dffb-cert\") pod \"ingress-canary-p5kfg\" (UID: \"26ff859f-ec36-43d8-888d-bba78cf3dffb\") " pod="openshift-ingress-canary/ingress-canary-p5kfg" Jul 15 19:22:07.655550 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:07.655233 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls podName:6c110945-616b-4146-ac13-c20e4c84ef1e nodeName:}" failed. No retries permitted until 2026-07-15 19:22:08.155209072 +0000 UTC m=+36.626684356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls") pod "dns-default-xtdvl" (UID: "6c110945-616b-4146-ac13-c20e4c84ef1e") : secret "dns-default-metrics-tls" not found Jul 15 19:22:07.655550 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.655398 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c110945-616b-4146-ac13-c20e4c84ef1e-tmp-dir\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:07.655786 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.655765 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c110945-616b-4146-ac13-c20e4c84ef1e-config-volume\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:07.660292 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.660268 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ff859f-ec36-43d8-888d-bba78cf3dffb-cert\") pod \"ingress-canary-p5kfg\" (UID: \"26ff859f-ec36-43d8-888d-bba78cf3dffb\") " pod="openshift-ingress-canary/ingress-canary-p5kfg" Jul 15 19:22:07.664897 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.664419 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9clf\" (UniqueName: \"kubernetes.io/projected/6c110945-616b-4146-ac13-c20e4c84ef1e-kube-api-access-r9clf\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:07.664897 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.664865 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2kgg\" (UniqueName: \"kubernetes.io/projected/26ff859f-ec36-43d8-888d-bba78cf3dffb-kube-api-access-f2kgg\") pod \"ingress-canary-p5kfg\" (UID: \"26ff859f-ec36-43d8-888d-bba78cf3dffb\") " pod="openshift-ingress-canary/ingress-canary-p5kfg" Jul 15 19:22:07.729941 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:07.729909 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p5kfg" Jul 15 19:22:08.054234 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:08.054139 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:22:08.054400 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:08.054149 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:22:08.054400 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:08.054157 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:22:08.057789 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:08.057371 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 19:22:08.057789 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:08.057420 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 19:22:08.057789 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:08.057444 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 19:22:08.057789 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:08.057626 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 19:22:08.057789 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:08.057657 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dmtm7\"" Jul 15 19:22:08.057789 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:08.057698 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jwgm9\"" Jul 15 19:22:08.158814 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:08.158777 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:08.159000 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:08.158925 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:22:08.159045 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:08.159003 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls podName:6c110945-616b-4146-ac13-c20e4c84ef1e nodeName:}" failed. No retries permitted until 2026-07-15 19:22:09.158987161 +0000 UTC m=+37.630462431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls") pod "dns-default-xtdvl" (UID: "6c110945-616b-4146-ac13-c20e4c84ef1e") : secret "dns-default-metrics-tls" not found Jul 15 19:22:09.166583 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:09.166544 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:09.167278 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:09.166679 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:22:09.167278 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:09.166743 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls podName:6c110945-616b-4146-ac13-c20e4c84ef1e nodeName:}" failed. No retries permitted until 2026-07-15 19:22:11.166723503 +0000 UTC m=+39.638198784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls") pod "dns-default-xtdvl" (UID: "6c110945-616b-4146-ac13-c20e4c84ef1e") : secret "dns-default-metrics-tls" not found Jul 15 19:22:09.369825 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:09.369570 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p5kfg"] Jul 15 19:22:09.372636 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:22:09.372612 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ff859f_ec36_43d8_888d_bba78cf3dffb.slice/crio-d5307c1aa7805b9f89d0c91878c0f5b72336fc79b68354f08d0884f0724d95ac WatchSource:0}: Error finding container d5307c1aa7805b9f89d0c91878c0f5b72336fc79b68354f08d0884f0724d95ac: Status 404 returned error can't find the container with id d5307c1aa7805b9f89d0c91878c0f5b72336fc79b68354f08d0884f0724d95ac Jul 15 19:22:09.554274 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:09.554241 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:09.615727 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:09.615695 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:10.315681 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:10.315647 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p5kfg" event={"ID":"26ff859f-ec36-43d8-888d-bba78cf3dffb","Type":"ContainerStarted","Data":"d5307c1aa7805b9f89d0c91878c0f5b72336fc79b68354f08d0884f0724d95ac"} Jul 15 19:22:10.319192 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:10.319161 2564 generic.go:356] "Generic (PLEG): container finished" podID="6fb28f74-8890-418f-97f3-6da7ba5bc4a8" containerID="a78aad5dc80abdd7f2468323aa9d26ac03d935bd8b92bd5237b7430291e41320" exitCode=0 Jul 15 19:22:10.319418 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:10.319224 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5blhm" event={"ID":"6fb28f74-8890-418f-97f3-6da7ba5bc4a8","Type":"ContainerDied","Data":"a78aad5dc80abdd7f2468323aa9d26ac03d935bd8b92bd5237b7430291e41320"} Jul 15 19:22:10.965589 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:10.965556 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n"] Jul 15 19:22:10.972316 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:10.972289 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:22:10.974702 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:10.974675 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 15 19:22:10.974819 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:10.974716 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 15 19:22:10.975222 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:10.975206 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 15 19:22:10.975304 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:10.975210 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 15 19:22:10.978419 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:10.978395 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n"] Jul 15 19:22:11.002389 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.002361 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62"] Jul 15 19:22:11.006154 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.006131 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.008567 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.008545 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 15 19:22:11.008662 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.008544 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 15 19:22:11.008702 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.008543 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 15 19:22:11.008747 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.008552 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 15 19:22:11.016450 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.016428 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62"] Jul 15 19:22:11.082734 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.082700 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e247c625-0e8f-40ab-84c1-57da96a312e4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.082734 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.082734 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e247c625-0e8f-40ab-84c1-57da96a312e4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.082967 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.082759 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxhgw\" (UniqueName: \"kubernetes.io/projected/e247c625-0e8f-40ab-84c1-57da96a312e4-kube-api-access-gxhgw\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.082967 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.082782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/502e08b3-045e-43f7-92dd-4e8173c71550-klusterlet-config\") pod \"klusterlet-addon-workmgr-f879444f9-z465n\" (UID: \"502e08b3-045e-43f7-92dd-4e8173c71550\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:22:11.082967 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.082918 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/502e08b3-045e-43f7-92dd-4e8173c71550-tmp\") pod \"klusterlet-addon-workmgr-f879444f9-z465n\" (UID: \"502e08b3-045e-43f7-92dd-4e8173c71550\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:22:11.082967 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.082958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e247c625-0e8f-40ab-84c1-57da96a312e4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.083164 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.083000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz5hn\" (UniqueName: \"kubernetes.io/projected/502e08b3-045e-43f7-92dd-4e8173c71550-kube-api-access-dz5hn\") pod \"klusterlet-addon-workmgr-f879444f9-z465n\" (UID: \"502e08b3-045e-43f7-92dd-4e8173c71550\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:22:11.083164 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.083031 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e247c625-0e8f-40ab-84c1-57da96a312e4-ca\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.083164 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.083054 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e247c625-0e8f-40ab-84c1-57da96a312e4-hub\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.184003 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.183961 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e247c625-0e8f-40ab-84c1-57da96a312e4-ca\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.184003 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.183999 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e247c625-0e8f-40ab-84c1-57da96a312e4-hub\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.184231 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.184044 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e247c625-0e8f-40ab-84c1-57da96a312e4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.184231 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.184070 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e247c625-0e8f-40ab-84c1-57da96a312e4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.184231 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.184087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxhgw\" (UniqueName: \"kubernetes.io/projected/e247c625-0e8f-40ab-84c1-57da96a312e4-kube-api-access-gxhgw\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.184231 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.184109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/502e08b3-045e-43f7-92dd-4e8173c71550-klusterlet-config\") pod \"klusterlet-addon-workmgr-f879444f9-z465n\" (UID: \"502e08b3-045e-43f7-92dd-4e8173c71550\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:22:11.184231 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.184139 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:11.184231 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.184203 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/502e08b3-045e-43f7-92dd-4e8173c71550-tmp\") pod \"klusterlet-addon-workmgr-f879444f9-z465n\" (UID: \"502e08b3-045e-43f7-92dd-4e8173c71550\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:22:11.184231 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.184228 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e247c625-0e8f-40ab-84c1-57da96a312e4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.184592 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.184259 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz5hn\" (UniqueName: \"kubernetes.io/projected/502e08b3-045e-43f7-92dd-4e8173c71550-kube-api-access-dz5hn\") pod \"klusterlet-addon-workmgr-f879444f9-z465n\" (UID: \"502e08b3-045e-43f7-92dd-4e8173c71550\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:22:11.184871 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:11.184837 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:22:11.184871 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.184867 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/502e08b3-045e-43f7-92dd-4e8173c71550-tmp\") pod \"klusterlet-addon-workmgr-f879444f9-z465n\" (UID: \"502e08b3-045e-43f7-92dd-4e8173c71550\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:22:11.185047 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:11.184949 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls podName:6c110945-616b-4146-ac13-c20e4c84ef1e nodeName:}" failed. No retries permitted until 2026-07-15 19:22:15.184916254 +0000 UTC m=+43.656391526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls") pod "dns-default-xtdvl" (UID: "6c110945-616b-4146-ac13-c20e4c84ef1e") : secret "dns-default-metrics-tls" not found Jul 15 19:22:11.185410 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.185362 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e247c625-0e8f-40ab-84c1-57da96a312e4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.188654 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.188582 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e247c625-0e8f-40ab-84c1-57da96a312e4-ca\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.188654 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.188605 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e247c625-0e8f-40ab-84c1-57da96a312e4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.188654 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.188622 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e247c625-0e8f-40ab-84c1-57da96a312e4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.189337 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.189313 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/502e08b3-045e-43f7-92dd-4e8173c71550-klusterlet-config\") pod \"klusterlet-addon-workmgr-f879444f9-z465n\" (UID: \"502e08b3-045e-43f7-92dd-4e8173c71550\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:22:11.190649 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.190629 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e247c625-0e8f-40ab-84c1-57da96a312e4-hub\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.192295 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.192272 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz5hn\" (UniqueName: \"kubernetes.io/projected/502e08b3-045e-43f7-92dd-4e8173c71550-kube-api-access-dz5hn\") pod \"klusterlet-addon-workmgr-f879444f9-z465n\" (UID: \"502e08b3-045e-43f7-92dd-4e8173c71550\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:22:11.192686 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.192663 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxhgw\" (UniqueName: \"kubernetes.io/projected/e247c625-0e8f-40ab-84c1-57da96a312e4-kube-api-access-gxhgw\") pod \"cluster-proxy-proxy-agent-7486df595f-plz62\" (UID: \"e247c625-0e8f-40ab-84c1-57da96a312e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.283201 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.283103 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:22:11.321580 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.321340 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:22:11.326144 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.326111 2564 generic.go:356] "Generic (PLEG): container finished" podID="6fb28f74-8890-418f-97f3-6da7ba5bc4a8" containerID="573eb59656a71f946aba07909d8b6c3d362b5e14a8473ca6f385d5f471b7e25e" exitCode=0 Jul 15 19:22:11.326294 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.326195 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5blhm" event={"ID":"6fb28f74-8890-418f-97f3-6da7ba5bc4a8","Type":"ContainerDied","Data":"573eb59656a71f946aba07909d8b6c3d362b5e14a8473ca6f385d5f471b7e25e"} Jul 15 19:22:11.422460 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.422424 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:11.485160 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.484981 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:11.569072 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.569042 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n"] Jul 15 19:22:11.573079 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:11.573058 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62"] Jul 15 19:22:11.702528 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:22:11.702495 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502e08b3_045e_43f7_92dd_4e8173c71550.slice/crio-b816779723bf2aec2160d59533ab9363999f7f6028ad26f552adbfa720702015 WatchSource:0}: Error finding container b816779723bf2aec2160d59533ab9363999f7f6028ad26f552adbfa720702015: Status 404 returned error can't find the container with id b816779723bf2aec2160d59533ab9363999f7f6028ad26f552adbfa720702015 Jul 15 19:22:11.703354 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:22:11.703329 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode247c625_0e8f_40ab_84c1_57da96a312e4.slice/crio-8555e5772c326db4855518e0757d93edc97ec367830aaf37e9f54f3f77aaa958 WatchSource:0}: Error finding container 8555e5772c326db4855518e0757d93edc97ec367830aaf37e9f54f3f77aaa958: Status 404 returned error can't find the container with id 8555e5772c326db4855518e0757d93edc97ec367830aaf37e9f54f3f77aaa958 Jul 15 19:22:12.329846 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:12.329809 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p5kfg" event={"ID":"26ff859f-ec36-43d8-888d-bba78cf3dffb","Type":"ContainerStarted","Data":"9782561f682beee55830da00a2009e966b9e9a09c1512f734ce884f80ab009b2"} Jul 15 19:22:12.330961 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:12.330941 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" event={"ID":"502e08b3-045e-43f7-92dd-4e8173c71550","Type":"ContainerStarted","Data":"b816779723bf2aec2160d59533ab9363999f7f6028ad26f552adbfa720702015"} Jul 15 19:22:12.332007 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:12.331979 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" event={"ID":"e247c625-0e8f-40ab-84c1-57da96a312e4","Type":"ContainerStarted","Data":"8555e5772c326db4855518e0757d93edc97ec367830aaf37e9f54f3f77aaa958"} Jul 15 19:22:12.335219 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:12.335196 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5blhm" event={"ID":"6fb28f74-8890-418f-97f3-6da7ba5bc4a8","Type":"ContainerStarted","Data":"2fb42d04b20a96261f9f5de2e2eeed795ec5545cba55791c684414ba462b636a"} Jul 15 19:22:12.344089 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:12.344052 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-p5kfg" podStartSLOduration=2.9836515439999998 podStartE2EDuration="5.344041241s" totalImagesPullingTime="2.360389697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:22:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:22:12.343430299 +0000 UTC m=+40.814905588" watchObservedRunningTime="2026-07-15 19:22:12.344041241 +0000 UTC m=+40.815516529" Jul 15 19:22:12.363500 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:12.363419 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5blhm" podStartSLOduration=7.607858275 podStartE2EDuration="39.363403301s" totalImagesPullingTime="31.755545026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:21:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:22:12.362690148 +0000 UTC m=+40.834165437" watchObservedRunningTime="2026-07-15 19:22:12.363403301 +0000 UTC m=+40.834878589" Jul 15 19:22:12.484417 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:12.484385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:12.540012 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:12.539925 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:12.683695 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:12.683656 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:12.782289 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:12.782249 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:15.223769 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:15.223736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:15.224214 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:15.223901 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:22:15.224214 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:15.223980 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls podName:6c110945-616b-4146-ac13-c20e4c84ef1e nodeName:}" failed. No retries permitted until 2026-07-15 19:22:23.223962125 +0000 UTC m=+51.695437411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls") pod "dns-default-xtdvl" (UID: "6c110945-616b-4146-ac13-c20e4c84ef1e") : secret "dns-default-metrics-tls" not found Jul 15 19:22:18.352705 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:18.352673 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" event={"ID":"502e08b3-045e-43f7-92dd-4e8173c71550","Type":"ContainerStarted","Data":"b7b0d49c4c127ef491858a497b91e018932d3a9c31cb53084b67e7cdd8dffca7"} Jul 15 19:22:18.353141 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:18.352811 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:22:18.354854 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:18.354829 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:22:18.355020 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:18.354997 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" event={"ID":"e247c625-0e8f-40ab-84c1-57da96a312e4","Type":"ContainerStarted","Data":"a392b9ccf4dba9dbd2f2dc33c7cb8f2527f51f915c63a5cce459ec36bcc50b0b"} Jul 15 19:22:18.355096 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:18.355036 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" event={"ID":"e247c625-0e8f-40ab-84c1-57da96a312e4","Type":"ContainerStarted","Data":"3b31cf42b2dbc02308bc7f7a19c927f9a33a9f5a1e74767b5be38df15f73b4bd"} Jul 15 19:22:18.355096 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:18.355054 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" event={"ID":"e247c625-0e8f-40ab-84c1-57da96a312e4","Type":"ContainerStarted","Data":"2bdf1266bbd89dbac05eec66de66e26364d47c45d4375b5b47c43ffe95345eff"} Jul 15 19:22:18.371303 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:18.371261 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" podStartSLOduration=2.197421948 podStartE2EDuration="8.371248934s" totalImagesPullingTime="6.173826986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:22:18.370428643 +0000 UTC m=+46.841903931" watchObservedRunningTime="2026-07-15 19:22:18.371248934 +0000 UTC m=+46.842724223" Jul 15 19:22:18.406976 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:18.406884 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" podStartSLOduration=2.246726319 podStartE2EDuration="8.40686884s" totalImagesPullingTime="6.160142521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:22:18.405841776 +0000 UTC m=+46.877317067" watchObservedRunningTime="2026-07-15 19:22:18.40686884 +0000 UTC m=+46.878344128" Jul 15 19:22:23.286082 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:23.286048 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:23.286598 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:23.286152 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:22:23.286598 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:23.286196 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls podName:6c110945-616b-4146-ac13-c20e4c84ef1e nodeName:}" failed. No retries permitted until 2026-07-15 19:22:39.286183444 +0000 UTC m=+67.757658712 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls") pod "dns-default-xtdvl" (UID: "6c110945-616b-4146-ac13-c20e4c84ef1e") : secret "dns-default-metrics-tls" not found Jul 15 19:22:26.408339 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:26.408303 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:26.463047 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:26.463014 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:32.396927 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:32.396895 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:32.458720 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:32.458691 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:34.306805 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:34.306780 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rr9n5" Jul 15 19:22:34.418031 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:34.417998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:34.477204 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:34.477173 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:36.670386 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:36.670329 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kdzv\" (UniqueName: \"kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv\") pod \"network-check-target-5w4r4\" (UID: \"5d03ee98-6e6f-4ee1-8b78-d4da30c503ff\") " pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:22:36.670386 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:36.670397 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs\") pod \"network-metrics-daemon-zfvrb\" (UID: \"1a59a892-2494-489b-b5f2-038c7161cf7e\") " pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:22:36.672812 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:36.672787 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 19:22:36.672921 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:36.672831 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 19:22:36.681394 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:36.681373 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 15 19:22:36.681492 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:36.681444 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs podName:1a59a892-2494-489b-b5f2-038c7161cf7e nodeName:}" failed. No retries permitted until 2026-07-15 19:23:40.681422635 +0000 UTC m=+129.152897903 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs") pod "network-metrics-daemon-zfvrb" (UID: "1a59a892-2494-489b-b5f2-038c7161cf7e") : secret "metrics-daemon-secret" not found Jul 15 19:22:36.683332 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:36.683313 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 19:22:36.695096 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:36.695067 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kdzv\" (UniqueName: \"kubernetes.io/projected/5d03ee98-6e6f-4ee1-8b78-d4da30c503ff-kube-api-access-5kdzv\") pod \"network-check-target-5w4r4\" (UID: \"5d03ee98-6e6f-4ee1-8b78-d4da30c503ff\") " pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:22:36.770740 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:36.770705 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret\") pod \"global-pull-secret-syncer-pz2h5\" (UID: \"e089ab9b-564c-4a0d-9b14-4bce14a4f070\") " pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:22:36.773141 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:36.773123 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 19:22:36.783959 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:36.783940 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e089ab9b-564c-4a0d-9b14-4bce14a4f070-original-pull-secret\") pod \"global-pull-secret-syncer-pz2h5\" (UID: \"e089ab9b-564c-4a0d-9b14-4bce14a4f070\") " pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:22:36.868874 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:36.868842 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jwgm9\"" Jul 15 19:22:36.872485 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:36.872454 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pz2h5" Jul 15 19:22:36.877292 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:36.877276 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:22:37.002574 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:37.002533 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pz2h5"] Jul 15 19:22:37.005976 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:22:37.005950 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode089ab9b_564c_4a0d_9b14_4bce14a4f070.slice/crio-80310af6a85ff9c60cc07f49797b1dd61465f6570bab383e2ae9a7a0d6e0abc1 WatchSource:0}: Error finding container 80310af6a85ff9c60cc07f49797b1dd61465f6570bab383e2ae9a7a0d6e0abc1: Status 404 returned error can't find the container with id 80310af6a85ff9c60cc07f49797b1dd61465f6570bab383e2ae9a7a0d6e0abc1 Jul 15 19:22:37.017835 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:37.017810 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5w4r4"] Jul 15 19:22:37.020445 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:22:37.020421 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d03ee98_6e6f_4ee1_8b78_d4da30c503ff.slice/crio-f3da857460f16743d4f1ab630ff3f28a5ed6ef11f99af0c0f2bd859d1bd7fdf9 WatchSource:0}: Error finding container f3da857460f16743d4f1ab630ff3f28a5ed6ef11f99af0c0f2bd859d1bd7fdf9: Status 404 returned error can't find the container with id f3da857460f16743d4f1ab630ff3f28a5ed6ef11f99af0c0f2bd859d1bd7fdf9 Jul 15 19:22:37.401928 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:37.401890 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5w4r4" event={"ID":"5d03ee98-6e6f-4ee1-8b78-d4da30c503ff","Type":"ContainerStarted","Data":"f3da857460f16743d4f1ab630ff3f28a5ed6ef11f99af0c0f2bd859d1bd7fdf9"} Jul 15 19:22:37.402906 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:37.402885 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pz2h5" event={"ID":"e089ab9b-564c-4a0d-9b14-4bce14a4f070","Type":"ContainerStarted","Data":"80310af6a85ff9c60cc07f49797b1dd61465f6570bab383e2ae9a7a0d6e0abc1"} Jul 15 19:22:39.286385 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:39.286351 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:22:39.286972 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:39.286542 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:22:39.286972 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:22:39.286626 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls podName:6c110945-616b-4146-ac13-c20e4c84ef1e nodeName:}" failed. No retries permitted until 2026-07-15 19:23:11.286605111 +0000 UTC m=+99.758080385 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls") pod "dns-default-xtdvl" (UID: "6c110945-616b-4146-ac13-c20e4c84ef1e") : secret "dns-default-metrics-tls" not found Jul 15 19:22:41.983109 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:41.983068 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:42.033406 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:42.033370 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:42.419868 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:42.419836 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pz2h5" event={"ID":"e089ab9b-564c-4a0d-9b14-4bce14a4f070","Type":"ContainerStarted","Data":"100585e1617acbdaec47a975f21fa59dd9e7add77d9c67da14b1511449ceee9b"} Jul 15 19:22:42.421372 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:42.421350 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5w4r4" event={"ID":"5d03ee98-6e6f-4ee1-8b78-d4da30c503ff","Type":"ContainerStarted","Data":"1180daf4382adc3fa047cbebbe787cf31ebf630cd449f613da82eaa3dba7c8e9"} Jul 15 19:22:42.421516 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:42.421490 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:22:42.433688 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:42.433646 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pz2h5" podStartSLOduration=64.665837374 podStartE2EDuration="1m9.433632282s" totalImagesPullingTime="4.767794908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:22:42.433216759 +0000 UTC m=+70.904692049" watchObservedRunningTime="2026-07-15 19:22:42.433632282 +0000 UTC m=+70.905107568" Jul 15 19:22:42.446933 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:22:42.446891 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5w4r4" podStartSLOduration=64.704307648 podStartE2EDuration="1m9.446877734s" totalImagesPullingTime="4.742570086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:22:42.446412038 +0000 UTC m=+70.917887317" watchObservedRunningTime="2026-07-15 19:22:42.446877734 +0000 UTC m=+70.918353022" Jul 15 19:23:04.782153 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:23:04.782020 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:04.836125 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:23:04.836084 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:11.294674 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:23:11.294633 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:23:11.295127 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:23:11.294792 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:23:11.295127 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:23:11.294865 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls podName:6c110945-616b-4146-ac13-c20e4c84ef1e nodeName:}" failed. No retries permitted until 2026-07-15 19:24:15.294848345 +0000 UTC m=+163.766323613 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls") pod "dns-default-xtdvl" (UID: "6c110945-616b-4146-ac13-c20e4c84ef1e") : secret "dns-default-metrics-tls" not found Jul 15 19:23:13.427635 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:23:13.427605 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5w4r4" Jul 15 19:23:40.704001 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:23:40.703966 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs\") pod \"network-metrics-daemon-zfvrb\" (UID: \"1a59a892-2494-489b-b5f2-038c7161cf7e\") " pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:23:40.704458 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:23:40.704085 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 15 19:23:40.704458 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:23:40.704137 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs podName:1a59a892-2494-489b-b5f2-038c7161cf7e nodeName:}" failed. No retries permitted until 2026-07-15 19:25:42.704123028 +0000 UTC m=+251.175598294 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs") pod "network-metrics-daemon-zfvrb" (UID: "1a59a892-2494-489b-b5f2-038c7161cf7e") : secret "metrics-daemon-secret" not found Jul 15 19:24:10.422904 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:24:10.422860 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xtdvl" podUID="6c110945-616b-4146-ac13-c20e4c84ef1e" Jul 15 19:24:10.671336 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:24:10.671306 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xtdvl" Jul 15 19:24:11.079295 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:24:11.079238 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:24:15.353530 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:24:15.353493 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:24:15.353903 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:24:15.353640 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:24:15.353903 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:24:15.353715 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls podName:6c110945-616b-4146-ac13-c20e4c84ef1e nodeName:}" failed. No retries permitted until 2026-07-15 19:26:17.353697117 +0000 UTC m=+285.825172384 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls") pod "dns-default-xtdvl" (UID: "6c110945-616b-4146-ac13-c20e4c84ef1e") : secret "dns-default-metrics-tls" not found Jul 15 19:24:18.353698 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:24:18.353599 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" podUID="502e08b3-045e-43f7-92dd-4e8173c71550" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.8:8000/readyz\": dial tcp 10.134.0.8:8000: connect: connection refused" Jul 15 19:24:18.694742 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:24:18.694662 2564 generic.go:356] "Generic (PLEG): container finished" podID="502e08b3-045e-43f7-92dd-4e8173c71550" containerID="b7b0d49c4c127ef491858a497b91e018932d3a9c31cb53084b67e7cdd8dffca7" exitCode=1 Jul 15 19:24:18.694878 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:24:18.694735 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" event={"ID":"502e08b3-045e-43f7-92dd-4e8173c71550","Type":"ContainerDied","Data":"b7b0d49c4c127ef491858a497b91e018932d3a9c31cb53084b67e7cdd8dffca7"} Jul 15 19:24:18.695061 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:24:18.695048 2564 scope.go:117] "RemoveContainer" containerID="b7b0d49c4c127ef491858a497b91e018932d3a9c31cb53084b67e7cdd8dffca7" Jul 15 19:24:18.745138 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:24:18.745109 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:24:19.699363 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:24:19.699330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" event={"ID":"502e08b3-045e-43f7-92dd-4e8173c71550","Type":"ContainerStarted","Data":"51b3c201e7232f9fdcab50ca51272828b865d2bf2965c29e0e155ecc59eb5e46"} Jul 15 19:24:19.699791 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:24:19.699620 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:24:19.700872 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:24:19.700851 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:24:26.054686 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:24:26.054604 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:25:35.584015 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:35.583973 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:35.665590 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:35.665559 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:37.862223 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:37.862184 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:37.963063 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:37.963031 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:40.144689 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.144648 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:40.460593 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.460569 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:40.566928 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.566896 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:40.648337 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.648301 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2gvbg"] Jul 15 19:25:40.650178 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.650163 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:40.658423 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.658402 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 19:25:40.658650 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.658638 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cld5n\"" Jul 15 19:25:40.659132 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.659118 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 19:25:40.661191 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.661178 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 19:25:40.661340 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.661326 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 19:25:40.668301 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.668277 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2gvbg"] Jul 15 19:25:40.679699 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.679678 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:40.730057 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.729988 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d91f01df-53be-4664-98ac-3a960fc0aebd-crio-socket\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:40.730057 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.730024 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d91f01df-53be-4664-98ac-3a960fc0aebd-data-volume\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:40.730057 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.730044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:40.730251 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.730075 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:40.730251 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.730103 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptjch\" (UniqueName: \"kubernetes.io/projected/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-api-access-ptjch\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:40.750079 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.750034 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:40.830782 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.830750 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptjch\" (UniqueName: \"kubernetes.io/projected/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-api-access-ptjch\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:40.830914 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.830808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d91f01df-53be-4664-98ac-3a960fc0aebd-crio-socket\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:40.830914 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.830833 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d91f01df-53be-4664-98ac-3a960fc0aebd-data-volume\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:40.830914 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.830849 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:40.830914 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.830876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:40.831124 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:40.830958 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:25:40.831124 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:40.830993 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:25:40.831124 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.830961 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d91f01df-53be-4664-98ac-3a960fc0aebd-crio-socket\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:40.831124 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:40.831033 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm podName:d91f01df-53be-4664-98ac-3a960fc0aebd nodeName:}" failed. No retries permitted until 2026-07-15 19:25:41.331014951 +0000 UTC m=+249.802490222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2gvbg" (UID: "d91f01df-53be-4664-98ac-3a960fc0aebd") : configmap "kube-rbac-proxy" not found Jul 15 19:25:40.831124 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:40.831051 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls podName:d91f01df-53be-4664-98ac-3a960fc0aebd nodeName:}" failed. No retries permitted until 2026-07-15 19:25:41.331041648 +0000 UTC m=+249.802516917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2gvbg" (UID: "d91f01df-53be-4664-98ac-3a960fc0aebd") : secret "insights-runtime-extractor-tls" not found Jul 15 19:25:40.831331 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.831207 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d91f01df-53be-4664-98ac-3a960fc0aebd-data-volume\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:40.852819 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:40.852789 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptjch\" (UniqueName: \"kubernetes.io/projected/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-api-access-ptjch\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:41.194457 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:41.194376 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:41.293378 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:41.293352 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:41.334847 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:41.334806 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:41.334847 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:41.334848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:41.335036 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:41.334914 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:25:41.335036 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:41.334952 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:25:41.335036 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:41.334979 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm podName:d91f01df-53be-4664-98ac-3a960fc0aebd nodeName:}" failed. No retries permitted until 2026-07-15 19:25:42.334963061 +0000 UTC m=+250.806438332 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2gvbg" (UID: "d91f01df-53be-4664-98ac-3a960fc0aebd") : configmap "kube-rbac-proxy" not found Jul 15 19:25:41.335036 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:41.334992 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls podName:d91f01df-53be-4664-98ac-3a960fc0aebd nodeName:}" failed. No retries permitted until 2026-07-15 19:25:42.334986639 +0000 UTC m=+250.806461906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2gvbg" (UID: "d91f01df-53be-4664-98ac-3a960fc0aebd") : secret "insights-runtime-extractor-tls" not found Jul 15 19:25:42.344437 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:42.344399 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:42.344437 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:42.344436 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:42.344920 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:42.344558 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:25:42.344920 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:42.344569 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:25:42.344920 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:42.344633 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls podName:d91f01df-53be-4664-98ac-3a960fc0aebd nodeName:}" failed. No retries permitted until 2026-07-15 19:25:44.344618833 +0000 UTC m=+252.816094102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2gvbg" (UID: "d91f01df-53be-4664-98ac-3a960fc0aebd") : secret "insights-runtime-extractor-tls" not found Jul 15 19:25:42.344920 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:42.344703 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm podName:d91f01df-53be-4664-98ac-3a960fc0aebd nodeName:}" failed. No retries permitted until 2026-07-15 19:25:44.344684314 +0000 UTC m=+252.816159584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2gvbg" (UID: "d91f01df-53be-4664-98ac-3a960fc0aebd") : configmap "kube-rbac-proxy" not found Jul 15 19:25:42.520234 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:42.520199 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:42.603781 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:42.603675 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:42.748598 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:42.748565 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs\") pod \"network-metrics-daemon-zfvrb\" (UID: \"1a59a892-2494-489b-b5f2-038c7161cf7e\") " pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:25:42.748734 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:42.748691 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 15 19:25:42.748783 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:42.748746 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs podName:1a59a892-2494-489b-b5f2-038c7161cf7e nodeName:}" failed. No retries permitted until 2026-07-15 19:27:44.748731292 +0000 UTC m=+373.220206559 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs") pod "network-metrics-daemon-zfvrb" (UID: "1a59a892-2494-489b-b5f2-038c7161cf7e") : secret "metrics-daemon-secret" not found Jul 15 19:25:42.920956 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:42.920862 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:43.021823 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:43.021791 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:43.416942 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:43.416915 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lrfwx_a323a5a4-1f3f-49b5-ae58-8b5c90592e5a/dns-node-resolver/0.log" Jul 15 19:25:43.550977 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:43.550944 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:44.317010 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:44.316965 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:44.362963 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:44.362939 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:44.363049 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:44.362969 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:44.363088 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:44.363060 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:25:44.363088 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:44.363062 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:25:44.363157 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:44.363120 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls podName:d91f01df-53be-4664-98ac-3a960fc0aebd nodeName:}" failed. No retries permitted until 2026-07-15 19:25:48.363105868 +0000 UTC m=+256.834581135 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2gvbg" (UID: "d91f01df-53be-4664-98ac-3a960fc0aebd") : secret "insights-runtime-extractor-tls" not found Jul 15 19:25:44.363157 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:44.363133 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm podName:d91f01df-53be-4664-98ac-3a960fc0aebd nodeName:}" failed. No retries permitted until 2026-07-15 19:25:48.363126466 +0000 UTC m=+256.834601733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2gvbg" (UID: "d91f01df-53be-4664-98ac-3a960fc0aebd") : configmap "kube-rbac-proxy" not found Jul 15 19:25:44.413683 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:44.413657 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n6mjl_98333cd6-ba91-4b38-bf05-64e47d2f0251/node-ca/0.log" Jul 15 19:25:44.515786 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:44.515746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:44.620886 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:44.620796 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:44.717618 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:44.717594 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:44.822311 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:44.822281 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:45.016806 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:45.016770 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p5kfg_26ff859f-ec36-43d8-888d-bba78cf3dffb/serve-healthcheck-canary/0.log" Jul 15 19:25:45.515065 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:45.515034 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:46.972502 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:46.972428 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:47.047390 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:47.047364 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:47.339546 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:47.339449 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:48.197540 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:48.197509 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:48.260422 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:48.260389 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:48.384666 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:48.384633 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:48.393709 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:48.393687 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:48.393797 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:48.393716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:48.393852 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:48.393821 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:25:48.393852 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:48.393829 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:25:48.393948 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:48.393896 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm podName:d91f01df-53be-4664-98ac-3a960fc0aebd nodeName:}" failed. No retries permitted until 2026-07-15 19:25:56.393875022 +0000 UTC m=+264.865350298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2gvbg" (UID: "d91f01df-53be-4664-98ac-3a960fc0aebd") : configmap "kube-rbac-proxy" not found Jul 15 19:25:48.393948 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:25:48.393915 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls podName:d91f01df-53be-4664-98ac-3a960fc0aebd nodeName:}" failed. No retries permitted until 2026-07-15 19:25:56.393905065 +0000 UTC m=+264.865380334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2gvbg" (UID: "d91f01df-53be-4664-98ac-3a960fc0aebd") : secret "insights-runtime-extractor-tls" not found Jul 15 19:25:48.627120 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:48.627070 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:48.797539 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:48.797509 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:48.994673 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:48.994642 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:49.056800 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:49.056769 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:49.991580 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:49.991553 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:50.192280 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:50.192175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:50.281981 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:50.281953 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:50.597644 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:50.597608 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:50.803234 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:50.803196 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:50.875005 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:50.874911 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:51.206223 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:51.206129 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:51.836049 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:51.836022 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:52.392145 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:52.392110 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:52.589840 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:52.589804 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:52.798564 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:52.798536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:52.988206 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:52.988175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:53.650315 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:53.650279 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:53.996489 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:53.996447 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:54.207706 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:54.207673 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:54.385831 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:54.385741 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:54.556437 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:54.556405 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:54.619741 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:54.619709 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:54.787114 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:54.787088 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:54.990702 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:54.990634 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:56.453595 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:56.453543 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:56.453595 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:56.453592 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:56.454067 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:56.454046 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d91f01df-53be-4664-98ac-3a960fc0aebd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:56.455982 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:56.455964 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d91f01df-53be-4664-98ac-3a960fc0aebd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2gvbg\" (UID: \"d91f01df-53be-4664-98ac-3a960fc0aebd\") " pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:56.559517 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:56.559458 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2gvbg" Jul 15 19:25:56.686641 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:56.686609 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2gvbg"] Jul 15 19:25:56.689438 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:25:56.689409 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91f01df_53be_4664_98ac_3a960fc0aebd.slice/crio-bb2f1a6af70b8413612d2c521d3995d7aa9d44f2c9ab659880f5524dd89816bc WatchSource:0}: Error finding container bb2f1a6af70b8413612d2c521d3995d7aa9d44f2c9ab659880f5524dd89816bc: Status 404 returned error can't find the container with id bb2f1a6af70b8413612d2c521d3995d7aa9d44f2c9ab659880f5524dd89816bc Jul 15 19:25:56.973053 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:25:56.973021 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2gvbg" event={"ID":"d91f01df-53be-4664-98ac-3a960fc0aebd","Type":"ContainerStarted","Data":"bb2f1a6af70b8413612d2c521d3995d7aa9d44f2c9ab659880f5524dd89816bc"} Jul 15 19:26:01.990165 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:01.990131 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2gvbg" event={"ID":"d91f01df-53be-4664-98ac-3a960fc0aebd","Type":"ContainerStarted","Data":"91909da2ae0fde984376a57e054193427ac04d21f6eabdf06319e17449de12f3"} Jul 15 19:26:02.722648 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:02.722610 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:02.825780 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:02.825741 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:02.897846 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:02.897810 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:03.005548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:03.005449 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:03.200462 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:03.200434 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:03.267217 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:03.267148 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:03.380940 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:03.380882 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:03.433337 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:03.433300 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:03.787220 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:03.787159 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:03.855861 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:03.855814 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:04.197325 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:04.197255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:04.298076 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:04.298033 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:04.399445 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:04.399412 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:05.022619 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:05.022592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:05.078755 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:05.078713 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:06.003780 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:06.003742 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2gvbg" event={"ID":"d91f01df-53be-4664-98ac-3a960fc0aebd","Type":"ContainerStarted","Data":"7b821683556981b271abb7a1fd58fff5b3659cc2c7f38973761e2765b45a3359"} Jul 15 19:26:06.846116 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:06.846082 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:06.957036 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:06.956985 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:09.019397 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:09.019360 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2gvbg" event={"ID":"d91f01df-53be-4664-98ac-3a960fc0aebd","Type":"ContainerStarted","Data":"3eac4cfca6a28491ad4d5b03d6de4c93459b9e16a86d1f6d6d0463aa157071a7"} Jul 15 19:26:09.043855 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:09.043809 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2gvbg" podStartSLOduration=17.263588715 podStartE2EDuration="29.0437949s" totalImagesPullingTime="11.780206185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:25:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:26:09.040600174 +0000 UTC m=+277.512075463" watchObservedRunningTime="2026-07-15 19:26:09.0437949 +0000 UTC m=+277.515270352" Jul 15 19:26:11.344485 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:11.344429 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:11.443205 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:11.443175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:11.522626 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:11.522586 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:11.624042 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:11.623957 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:11.902973 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:11.902889 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:13.631186 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:13.631140 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:13.672202 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:26:13.672164 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xtdvl" podUID="6c110945-616b-4146-ac13-c20e4c84ef1e" Jul 15 19:26:13.687382 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:13.687358 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:14.033100 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:14.033070 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xtdvl" Jul 15 19:26:16.153198 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:16.153170 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:16.225004 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:16.224972 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:16.560688 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:16.560657 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:16.632296 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:16.632261 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:16.686833 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:16.686803 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:16.979632 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:16.979603 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:17.413250 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:17.413173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:26:17.415602 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:17.415581 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c110945-616b-4146-ac13-c20e4c84ef1e-metrics-tls\") pod \"dns-default-xtdvl\" (UID: \"6c110945-616b-4146-ac13-c20e4c84ef1e\") " pod="openshift-dns/dns-default-xtdvl" Jul 15 19:26:17.636964 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:17.636937 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bw2gg\"" Jul 15 19:26:17.644562 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:17.644542 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xtdvl" Jul 15 19:26:17.775045 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:17.775021 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xtdvl"] Jul 15 19:26:17.777147 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:26:17.777116 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c110945_616b_4146_ac13_c20e4c84ef1e.slice/crio-63f0165276b4bdd4eafb099c06762c5fd033eaccaaa104bd4a954bf9260094a0 WatchSource:0}: Error finding container 63f0165276b4bdd4eafb099c06762c5fd033eaccaaa104bd4a954bf9260094a0: Status 404 returned error can't find the container with id 63f0165276b4bdd4eafb099c06762c5fd033eaccaaa104bd4a954bf9260094a0 Jul 15 19:26:18.045101 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:18.045067 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xtdvl" event={"ID":"6c110945-616b-4146-ac13-c20e4c84ef1e","Type":"ContainerStarted","Data":"63f0165276b4bdd4eafb099c06762c5fd033eaccaaa104bd4a954bf9260094a0"} Jul 15 19:26:19.050702 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:19.050614 2564 generic.go:356] "Generic (PLEG): container finished" podID="502e08b3-045e-43f7-92dd-4e8173c71550" containerID="51b3c201e7232f9fdcab50ca51272828b865d2bf2965c29e0e155ecc59eb5e46" exitCode=1 Jul 15 19:26:19.050702 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:19.050656 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" event={"ID":"502e08b3-045e-43f7-92dd-4e8173c71550","Type":"ContainerDied","Data":"51b3c201e7232f9fdcab50ca51272828b865d2bf2965c29e0e155ecc59eb5e46"} Jul 15 19:26:19.051203 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:19.050722 2564 scope.go:117] "RemoveContainer" containerID="b7b0d49c4c127ef491858a497b91e018932d3a9c31cb53084b67e7cdd8dffca7" Jul 15 19:26:19.051203 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:19.051050 2564 scope.go:117] "RemoveContainer" containerID="51b3c201e7232f9fdcab50ca51272828b865d2bf2965c29e0e155ecc59eb5e46" Jul 15 19:26:19.051309 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:26:19.051235 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-f879444f9-z465n_open-cluster-management-agent-addon(502e08b3-045e-43f7-92dd-4e8173c71550)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" podUID="502e08b3-045e-43f7-92dd-4e8173c71550" Jul 15 19:26:19.699747 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:19.699635 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:26:20.055114 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:20.055089 2564 scope.go:117] "RemoveContainer" containerID="51b3c201e7232f9fdcab50ca51272828b865d2bf2965c29e0e155ecc59eb5e46" Jul 15 19:26:20.055573 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:26:20.055257 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-f879444f9-z465n_open-cluster-management-agent-addon(502e08b3-045e-43f7-92dd-4e8173c71550)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" podUID="502e08b3-045e-43f7-92dd-4e8173c71550" Jul 15 19:26:20.057369 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:20.057348 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xtdvl" Jul 15 19:26:20.057484 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:20.057397 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xtdvl" event={"ID":"6c110945-616b-4146-ac13-c20e4c84ef1e","Type":"ContainerStarted","Data":"7d2e9ccfc2de970025198cd3b114ffd07d9983f356576eeb444a943ecad1221d"} Jul 15 19:26:20.057484 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:20.057421 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xtdvl" event={"ID":"6c110945-616b-4146-ac13-c20e4c84ef1e","Type":"ContainerStarted","Data":"49ecd1ae6cebb945db04faaa695a0e40c583037340c56291e5937e3989c53b47"} Jul 15 19:26:20.106070 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:20.106014 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xtdvl" podStartSLOduration=251.59950672 podStartE2EDuration="4m13.10599653s" totalImagesPullingTime="1.50648981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:22:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:26:20.105866461 +0000 UTC m=+288.577341749" watchObservedRunningTime="2026-07-15 19:26:20.10599653 +0000 UTC m=+288.577471821" Jul 15 19:26:20.470504 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:20.470457 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:21.283978 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:21.283952 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:26:21.284332 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:21.284279 2564 scope.go:117] "RemoveContainer" containerID="51b3c201e7232f9fdcab50ca51272828b865d2bf2965c29e0e155ecc59eb5e46" Jul 15 19:26:21.284438 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:26:21.284419 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-f879444f9-z465n_open-cluster-management-agent-addon(502e08b3-045e-43f7-92dd-4e8173c71550)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" podUID="502e08b3-045e-43f7-92dd-4e8173c71550" Jul 15 19:26:21.323387 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:21.323341 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" podUID="e247c625-0e8f-40ab-84c1-57da96a312e4" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 15 19:26:22.274874 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:22.274816 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:22.363609 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:22.363579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:27.538709 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:27.538663 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:29.055915 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:26:29.055873 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zfvrb" podUID="1a59a892-2494-489b-b5f2-038c7161cf7e" Jul 15 19:26:29.157085 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:29.157052 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:29.259671 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:29.259641 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:29.775602 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:29.775569 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:29.986941 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:29.986906 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:30.061904 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.061832 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xtdvl" Jul 15 19:26:30.109119 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.109092 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:30.131811 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.131783 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6gpmh"] Jul 15 19:26:30.134825 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.134809 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.137919 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.137899 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 19:26:30.138099 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.138088 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 19:26:30.138495 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.138461 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 19:26:30.138629 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.138609 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qvh6l\"" Jul 15 19:26:30.138976 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.138961 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 19:26:30.139441 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.139427 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 19:26:30.139563 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.139545 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 19:26:30.201057 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.201022 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:30.205128 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.205105 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/681ecdbc-c87a-4a18-b17d-af54d2268c06-root\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.205242 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.205140 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/681ecdbc-c87a-4a18-b17d-af54d2268c06-node-exporter-textfile\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.205242 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.205165 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/681ecdbc-c87a-4a18-b17d-af54d2268c06-node-exporter-tls\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.205242 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.205189 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/681ecdbc-c87a-4a18-b17d-af54d2268c06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.205242 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.205212 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss9fv\" (UniqueName: \"kubernetes.io/projected/681ecdbc-c87a-4a18-b17d-af54d2268c06-kube-api-access-ss9fv\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.205398 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.205256 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/681ecdbc-c87a-4a18-b17d-af54d2268c06-node-exporter-accelerators-collector-config\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.205398 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.205272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/681ecdbc-c87a-4a18-b17d-af54d2268c06-node-exporter-wtmp\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.205398 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.205302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/681ecdbc-c87a-4a18-b17d-af54d2268c06-metrics-client-ca\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.205398 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.205335 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/681ecdbc-c87a-4a18-b17d-af54d2268c06-sys\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.297085 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.297059 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:30.305996 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.305971 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/681ecdbc-c87a-4a18-b17d-af54d2268c06-node-exporter-accelerators-collector-config\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.306117 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.306007 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/681ecdbc-c87a-4a18-b17d-af54d2268c06-node-exporter-wtmp\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.306117 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.306037 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/681ecdbc-c87a-4a18-b17d-af54d2268c06-metrics-client-ca\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.306117 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.306060 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/681ecdbc-c87a-4a18-b17d-af54d2268c06-sys\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.306117 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.306087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/681ecdbc-c87a-4a18-b17d-af54d2268c06-root\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.306312 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.306116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/681ecdbc-c87a-4a18-b17d-af54d2268c06-node-exporter-textfile\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.306312 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.306145 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/681ecdbc-c87a-4a18-b17d-af54d2268c06-node-exporter-tls\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.306312 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.306165 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/681ecdbc-c87a-4a18-b17d-af54d2268c06-sys\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.306312 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.306188 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/681ecdbc-c87a-4a18-b17d-af54d2268c06-root\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.306312 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.306170 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/681ecdbc-c87a-4a18-b17d-af54d2268c06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.306312 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.306247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss9fv\" (UniqueName: \"kubernetes.io/projected/681ecdbc-c87a-4a18-b17d-af54d2268c06-kube-api-access-ss9fv\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.306312 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.306165 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/681ecdbc-c87a-4a18-b17d-af54d2268c06-node-exporter-wtmp\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.306669 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.306598 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/681ecdbc-c87a-4a18-b17d-af54d2268c06-node-exporter-textfile\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.306753 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.306730 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/681ecdbc-c87a-4a18-b17d-af54d2268c06-node-exporter-accelerators-collector-config\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.306795 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.306756 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/681ecdbc-c87a-4a18-b17d-af54d2268c06-metrics-client-ca\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.308879 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.308859 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/681ecdbc-c87a-4a18-b17d-af54d2268c06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.308879 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.308867 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/681ecdbc-c87a-4a18-b17d-af54d2268c06-node-exporter-tls\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.318401 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.318345 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss9fv\" (UniqueName: \"kubernetes.io/projected/681ecdbc-c87a-4a18-b17d-af54d2268c06-kube-api-access-ss9fv\") pod \"node-exporter-6gpmh\" (UID: \"681ecdbc-c87a-4a18-b17d-af54d2268c06\") " pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.444025 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.443986 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6gpmh" Jul 15 19:26:30.454605 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:26:30.454581 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681ecdbc_c87a_4a18_b17d_af54d2268c06.slice/crio-f6f935b8780df693962d6d0e08c829b8d998580a6ba7bd89f1ff70b2f91b3bcc WatchSource:0}: Error finding container f6f935b8780df693962d6d0e08c829b8d998580a6ba7bd89f1ff70b2f91b3bcc: Status 404 returned error can't find the container with id f6f935b8780df693962d6d0e08c829b8d998580a6ba7bd89f1ff70b2f91b3bcc Jul 15 19:26:30.457399 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.457376 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:30.927430 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:30.927393 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:31.026071 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.026046 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:31.088522 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.088487 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6gpmh" event={"ID":"681ecdbc-c87a-4a18-b17d-af54d2268c06","Type":"ContainerStarted","Data":"f6f935b8780df693962d6d0e08c829b8d998580a6ba7bd89f1ff70b2f91b3bcc"} Jul 15 19:26:31.137366 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.137319 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:31.143821 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.143797 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:26:31.147772 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.147752 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.153697 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.153674 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 19:26:31.154423 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.154403 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 19:26:31.154972 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.154952 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 19:26:31.155061 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.154992 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 19:26:31.155121 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.155055 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 19:26:31.155274 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.155258 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 19:26:31.155358 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.155334 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 19:26:31.155546 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.155444 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 19:26:31.155683 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.155668 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8gs8f\"" Jul 15 19:26:31.158084 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.158067 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 19:26:31.166567 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.166547 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:26:31.213010 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.212978 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.213142 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.213024 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.213142 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.213057 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97758987-621f-4e7d-acaf-4ce87bd484f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.213142 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.213083 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.213142 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.213124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-web-config\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.213322 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.213145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97758987-621f-4e7d-acaf-4ce87bd484f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.213322 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.213175 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97758987-621f-4e7d-acaf-4ce87bd484f0-config-out\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.213322 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.213199 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc2mz\" (UniqueName: \"kubernetes.io/projected/97758987-621f-4e7d-acaf-4ce87bd484f0-kube-api-access-gc2mz\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.213322 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.213258 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97758987-621f-4e7d-acaf-4ce87bd484f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.213532 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.213328 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.213532 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.213363 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.213532 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.213396 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97758987-621f-4e7d-acaf-4ce87bd484f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.213532 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.213433 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.243080 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.243054 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:31.313981 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.313955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.313981 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.313984 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.314206 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.314003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97758987-621f-4e7d-acaf-4ce87bd484f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.314206 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.314021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.314206 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.314171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.314206 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.314202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.314410 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.314226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97758987-621f-4e7d-acaf-4ce87bd484f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.314410 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:26:31.314310 2564 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 15 19:26:31.314410 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.314344 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.314410 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:26:31.314370 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-main-tls podName:97758987-621f-4e7d-acaf-4ce87bd484f0 nodeName:}" failed. No retries permitted until 2026-07-15 19:26:31.814349216 +0000 UTC m=+300.285824506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "97758987-621f-4e7d-acaf-4ce87bd484f0") : secret "alertmanager-main-tls" not found Jul 15 19:26:31.314673 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.314427 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-web-config\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.314673 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.314452 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97758987-621f-4e7d-acaf-4ce87bd484f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.314673 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.314512 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97758987-621f-4e7d-acaf-4ce87bd484f0-config-out\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.314673 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.314536 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2mz\" (UniqueName: \"kubernetes.io/projected/97758987-621f-4e7d-acaf-4ce87bd484f0-kube-api-access-gc2mz\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.314673 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.314566 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97758987-621f-4e7d-acaf-4ce87bd484f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.315408 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.315125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97758987-621f-4e7d-acaf-4ce87bd484f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.315789 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.315718 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97758987-621f-4e7d-acaf-4ce87bd484f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.316621 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.316580 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97758987-621f-4e7d-acaf-4ce87bd484f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.317180 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.317155 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97758987-621f-4e7d-acaf-4ce87bd484f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.317849 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.317788 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.318083 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.318055 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.318355 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.318325 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.318785 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.318659 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-web-config\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.318785 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.318703 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97758987-621f-4e7d-acaf-4ce87bd484f0-config-out\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.318785 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.318739 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.319600 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.319581 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.322636 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.322608 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" podUID="e247c625-0e8f-40ab-84c1-57da96a312e4" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 15 19:26:31.324750 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.324732 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc2mz\" (UniqueName: \"kubernetes.io/projected/97758987-621f-4e7d-acaf-4ce87bd484f0-kube-api-access-gc2mz\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.512374 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.512302 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:31.819048 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.818954 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.821395 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.821368 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:31.883443 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:31.883420 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 19:26:32.061023 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:32.060906 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8gs8f\"" Jul 15 19:26:32.080822 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:32.069487 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:26:32.097023 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:32.096992 2564 generic.go:356] "Generic (PLEG): container finished" podID="681ecdbc-c87a-4a18-b17d-af54d2268c06" containerID="331cbf38683ec58c9a9a8747e18fef5624c2623a152bbd7d550098b5b90a2745" exitCode=0 Jul 15 19:26:32.097462 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:32.097036 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6gpmh" event={"ID":"681ecdbc-c87a-4a18-b17d-af54d2268c06","Type":"ContainerDied","Data":"331cbf38683ec58c9a9a8747e18fef5624c2623a152bbd7d550098b5b90a2745"} Jul 15 19:26:32.219609 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:32.219557 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:26:32.226618 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:26:32.226588 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97758987_621f_4e7d_acaf_4ce87bd484f0.slice/crio-6a0dcb3438b4f139ec9b3532cdb7ba501a6f57aa677f76646a746d110923280b WatchSource:0}: Error finding container 6a0dcb3438b4f139ec9b3532cdb7ba501a6f57aa677f76646a746d110923280b: Status 404 returned error can't find the container with id 6a0dcb3438b4f139ec9b3532cdb7ba501a6f57aa677f76646a746d110923280b Jul 15 19:26:32.648140 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:32.648101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:32.748027 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:32.747997 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:33.102163 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:33.102129 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6gpmh" event={"ID":"681ecdbc-c87a-4a18-b17d-af54d2268c06","Type":"ContainerStarted","Data":"ab46ce108827aeb676d224928f7213d1fbaf17182ca04c444359b8fc472dd355"} Jul 15 19:26:33.102623 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:33.102196 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6gpmh" event={"ID":"681ecdbc-c87a-4a18-b17d-af54d2268c06","Type":"ContainerStarted","Data":"98f026a9dd002997746e9ce4d90cedf92edabb1bee0a0f00b1b70546add48e4c"} Jul 15 19:26:33.103383 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:33.103361 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97758987-621f-4e7d-acaf-4ce87bd484f0","Type":"ContainerStarted","Data":"6a0dcb3438b4f139ec9b3532cdb7ba501a6f57aa677f76646a746d110923280b"} Jul 15 19:26:33.114717 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:33.114692 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:33.129038 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:33.128993 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6gpmh" podStartSLOduration=1.9672342409999999 podStartE2EDuration="3.128980162s" totalImagesPullingTime="1.091803119s" totalInitContainerRuntime="69.942802ms" isStatefulPod=true podCreationTimestamp="2026-07-15 19:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:26:33.12740455 +0000 UTC m=+301.598879852" watchObservedRunningTime="2026-07-15 19:26:33.128980162 +0000 UTC m=+301.600455485" Jul 15 19:26:33.207165 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:33.207128 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:33.297266 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:33.297228 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:33.653403 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:33.653371 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:34.108057 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:34.108026 2564 generic.go:356] "Generic (PLEG): container finished" podID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerID="7509db4546e472910d011180a4dbe47c867495656e90340854de30053941eefd" exitCode=0 Jul 15 19:26:34.108426 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:34.108111 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97758987-621f-4e7d-acaf-4ce87bd484f0","Type":"ContainerDied","Data":"7509db4546e472910d011180a4dbe47c867495656e90340854de30053941eefd"} Jul 15 19:26:34.109141 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:34.109127 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:26:34.570730 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:34.570699 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:34.630242 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:34.630204 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:34.803506 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:34.803454 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:34.912420 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:34.912353 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:34.999717 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:34.999658 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:35.054255 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:35.054224 2564 scope.go:117] "RemoveContainer" containerID="51b3c201e7232f9fdcab50ca51272828b865d2bf2965c29e0e155ecc59eb5e46" Jul 15 19:26:35.100549 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:35.100518 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:35.202558 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:35.202297 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:36.116731 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:36.116698 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" event={"ID":"502e08b3-045e-43f7-92dd-4e8173c71550","Type":"ContainerStarted","Data":"ac4a2b981c05071de6135f35d0fcb6b5e3bc84f2d14cb4502bd417a864150dec"} Jul 15 19:26:36.117012 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:36.116986 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:26:36.118323 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:36.118305 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f879444f9-z465n" Jul 15 19:26:36.529822 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:36.529793 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:36.623701 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:36.623666 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:36.731157 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:36.731128 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:36.835334 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:36.835308 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:37.122672 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:37.122636 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97758987-621f-4e7d-acaf-4ce87bd484f0","Type":"ContainerStarted","Data":"e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f"} Jul 15 19:26:37.122824 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:37.122709 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97758987-621f-4e7d-acaf-4ce87bd484f0","Type":"ContainerStarted","Data":"36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743"} Jul 15 19:26:37.122824 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:37.122746 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97758987-621f-4e7d-acaf-4ce87bd484f0","Type":"ContainerStarted","Data":"91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871"} Jul 15 19:26:37.122824 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:37.122772 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97758987-621f-4e7d-acaf-4ce87bd484f0","Type":"ContainerStarted","Data":"f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a"} Jul 15 19:26:37.122824 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:37.122794 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97758987-621f-4e7d-acaf-4ce87bd484f0","Type":"ContainerStarted","Data":"8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233"} Jul 15 19:26:38.129192 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:38.129158 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97758987-621f-4e7d-acaf-4ce87bd484f0","Type":"ContainerStarted","Data":"8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a"} Jul 15 19:26:38.160133 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:38.160088 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.67505057 podStartE2EDuration="7.160073427s" totalImagesPullingTime="4.46798274s" totalInitContainerRuntime="17.040117ms" isStatefulPod=false podCreationTimestamp="2026-07-15 19:26:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:26:38.158838305 +0000 UTC m=+306.630313594" watchObservedRunningTime="2026-07-15 19:26:38.160073427 +0000 UTC m=+306.631548715" Jul 15 19:26:40.054458 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:40.054418 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:26:41.322864 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:41.322824 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" podUID="e247c625-0e8f-40ab-84c1-57da96a312e4" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 15 19:26:41.323291 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:41.322893 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" Jul 15 19:26:41.323351 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:41.323328 2564 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"a392b9ccf4dba9dbd2f2dc33c7cb8f2527f51f915c63a5cce459ec36bcc50b0b"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 15 19:26:41.323395 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:41.323381 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" podUID="e247c625-0e8f-40ab-84c1-57da96a312e4" containerName="service-proxy" containerID="cri-o://a392b9ccf4dba9dbd2f2dc33c7cb8f2527f51f915c63a5cce459ec36bcc50b0b" gracePeriod=30 Jul 15 19:26:42.144120 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:42.144080 2564 generic.go:356] "Generic (PLEG): container finished" podID="e247c625-0e8f-40ab-84c1-57da96a312e4" containerID="a392b9ccf4dba9dbd2f2dc33c7cb8f2527f51f915c63a5cce459ec36bcc50b0b" exitCode=2 Jul 15 19:26:42.144283 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:42.144137 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" event={"ID":"e247c625-0e8f-40ab-84c1-57da96a312e4","Type":"ContainerDied","Data":"a392b9ccf4dba9dbd2f2dc33c7cb8f2527f51f915c63a5cce459ec36bcc50b0b"} Jul 15 19:26:42.144283 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:42.144172 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7486df595f-plz62" event={"ID":"e247c625-0e8f-40ab-84c1-57da96a312e4","Type":"ContainerStarted","Data":"d510fb4dd0cdf75eaa66f944f462def3a7907b24c4cd1ca45528ef531672778f"} Jul 15 19:26:42.307012 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:42.306978 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:43.085244 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:43.085179 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:43.142636 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:43.142609 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:44.150900 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:44.150854 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:45.691625 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:45.691589 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:45.743895 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:45.743835 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:53.170596 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:53.170561 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:53.263220 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:53.263175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:53.361747 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:53.361703 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:53.461647 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:53.461606 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:54.579485 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:54.579430 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:54.677834 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:26:54.677807 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:09.594233 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:09.594203 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:09.664249 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:09.664215 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:11.017595 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:11.017563 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:16.541643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:16.541609 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:16.578251 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:16.578226 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p5kfg_26ff859f-ec36-43d8-888d-bba78cf3dffb/serve-healthcheck-canary/0.log" Jul 15 19:27:16.595292 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:16.595220 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:16.879873 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:16.879787 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:16.979004 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:16.978932 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:17.086512 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:17.086457 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:42.403407 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:42.403325 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:42.507047 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:42.507024 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:44.765313 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:44.765281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs\") pod \"network-metrics-daemon-zfvrb\" (UID: \"1a59a892-2494-489b-b5f2-038c7161cf7e\") " pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:27:44.767662 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:44.767643 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59a892-2494-489b-b5f2-038c7161cf7e-metrics-certs\") pod \"network-metrics-daemon-zfvrb\" (UID: \"1a59a892-2494-489b-b5f2-038c7161cf7e\") " pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:27:44.857852 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:44.857820 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dmtm7\"" Jul 15 19:27:44.866388 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:44.866370 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfvrb" Jul 15 19:27:44.997602 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:44.997449 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zfvrb"] Jul 15 19:27:45.000229 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:27:45.000200 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a59a892_2494_489b_b5f2_038c7161cf7e.slice/crio-a7d16e1811ee55678332e98d5a9bb8c23b740d7afcf234bfc6bddf261ac97b4a WatchSource:0}: Error finding container a7d16e1811ee55678332e98d5a9bb8c23b740d7afcf234bfc6bddf261ac97b4a: Status 404 returned error can't find the container with id a7d16e1811ee55678332e98d5a9bb8c23b740d7afcf234bfc6bddf261ac97b4a Jul 15 19:27:45.349312 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:45.349221 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zfvrb" event={"ID":"1a59a892-2494-489b-b5f2-038c7161cf7e","Type":"ContainerStarted","Data":"a7d16e1811ee55678332e98d5a9bb8c23b740d7afcf234bfc6bddf261ac97b4a"} Jul 15 19:27:46.296388 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:46.296339 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:46.352685 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:46.352656 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:47.357897 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:47.357860 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zfvrb" event={"ID":"1a59a892-2494-489b-b5f2-038c7161cf7e","Type":"ContainerStarted","Data":"a56e5d55f0880d259f20e35b58bae370c5bf9a6c9c23d98d6da5d8a5a02c5310"} Jul 15 19:27:47.358319 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:47.357913 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zfvrb" event={"ID":"1a59a892-2494-489b-b5f2-038c7161cf7e","Type":"ContainerStarted","Data":"4c18826fc3b48297375478adbb64705fd81ce615bd759ed6120c5143f8a728ae"} Jul 15 19:27:47.377674 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:47.377626 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zfvrb" podStartSLOduration=374.019600797 podStartE2EDuration="6m15.37761326s" totalImagesPullingTime="1.358012463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:21:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:27:47.375705978 +0000 UTC m=+375.847181267" watchObservedRunningTime="2026-07-15 19:27:47.37761326 +0000 UTC m=+375.849088548" Jul 15 19:27:47.483768 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:47.483731 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:47.582647 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:47.582614 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:48.085238 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:48.085194 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:48.135712 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:48.135685 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:50.585530 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:50.585491 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:27:50.585985 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:50.585860 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="alertmanager" containerID="cri-o://8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233" gracePeriod=120 Jul 15 19:27:50.585985 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:50.585914 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="kube-rbac-proxy-metric" containerID="cri-o://e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f" gracePeriod=120 Jul 15 19:27:50.585985 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:50.585932 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="kube-rbac-proxy-web" containerID="cri-o://91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871" gracePeriod=120 Jul 15 19:27:50.585985 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:50.585956 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="prom-label-proxy" containerID="cri-o://8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a" gracePeriod=120 Jul 15 19:27:50.586200 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:50.585934 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="kube-rbac-proxy" containerID="cri-o://36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743" gracePeriod=120 Jul 15 19:27:50.586200 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:50.585998 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="config-reloader" containerID="cri-o://f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a" gracePeriod=120 Jul 15 19:27:51.375004 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.374968 2564 generic.go:356] "Generic (PLEG): container finished" podID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerID="8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a" exitCode=0 Jul 15 19:27:51.375004 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.374999 2564 generic.go:356] "Generic (PLEG): container finished" podID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerID="36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743" exitCode=0 Jul 15 19:27:51.375004 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.375009 2564 generic.go:356] "Generic (PLEG): container finished" podID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerID="f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a" exitCode=0 Jul 15 19:27:51.375237 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.375016 2564 generic.go:356] "Generic (PLEG): container finished" podID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerID="8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233" exitCode=0 Jul 15 19:27:51.375237 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.375039 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97758987-621f-4e7d-acaf-4ce87bd484f0","Type":"ContainerDied","Data":"8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a"} Jul 15 19:27:51.375237 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.375143 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97758987-621f-4e7d-acaf-4ce87bd484f0","Type":"ContainerDied","Data":"36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743"} Jul 15 19:27:51.375237 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.375175 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97758987-621f-4e7d-acaf-4ce87bd484f0","Type":"ContainerDied","Data":"f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a"} Jul 15 19:27:51.375237 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.375202 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97758987-621f-4e7d-acaf-4ce87bd484f0","Type":"ContainerDied","Data":"8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233"} Jul 15 19:27:51.832904 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.832879 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:51.922541 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.922444 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-web-config\") pod \"97758987-621f-4e7d-acaf-4ce87bd484f0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " Jul 15 19:27:51.922541 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.922491 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97758987-621f-4e7d-acaf-4ce87bd484f0-config-out\") pod \"97758987-621f-4e7d-acaf-4ce87bd484f0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " Jul 15 19:27:51.922541 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.922513 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-cluster-tls-config\") pod \"97758987-621f-4e7d-acaf-4ce87bd484f0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " Jul 15 19:27:51.922541 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.922537 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97758987-621f-4e7d-acaf-4ce87bd484f0-metrics-client-ca\") pod \"97758987-621f-4e7d-acaf-4ce87bd484f0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " Jul 15 19:27:51.922875 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.922569 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gc2mz\" (UniqueName: \"kubernetes.io/projected/97758987-621f-4e7d-acaf-4ce87bd484f0-kube-api-access-gc2mz\") pod \"97758987-621f-4e7d-acaf-4ce87bd484f0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " Jul 15 19:27:51.922875 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.922586 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97758987-621f-4e7d-acaf-4ce87bd484f0-alertmanager-main-db\") pod \"97758987-621f-4e7d-acaf-4ce87bd484f0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " Jul 15 19:27:51.922875 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.922601 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"97758987-621f-4e7d-acaf-4ce87bd484f0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " Jul 15 19:27:51.922875 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.922620 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97758987-621f-4e7d-acaf-4ce87bd484f0-alertmanager-trusted-ca-bundle\") pod \"97758987-621f-4e7d-acaf-4ce87bd484f0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " Jul 15 19:27:51.922875 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.922640 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-main-tls\") pod \"97758987-621f-4e7d-acaf-4ce87bd484f0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " Jul 15 19:27:51.922875 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.922663 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"97758987-621f-4e7d-acaf-4ce87bd484f0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " Jul 15 19:27:51.922875 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.922692 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy\") pod \"97758987-621f-4e7d-acaf-4ce87bd484f0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " Jul 15 19:27:51.922875 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.922730 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-config-volume\") pod \"97758987-621f-4e7d-acaf-4ce87bd484f0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " Jul 15 19:27:51.922875 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.922773 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97758987-621f-4e7d-acaf-4ce87bd484f0-tls-assets\") pod \"97758987-621f-4e7d-acaf-4ce87bd484f0\" (UID: \"97758987-621f-4e7d-acaf-4ce87bd484f0\") " Jul 15 19:27:51.923490 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.923020 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97758987-621f-4e7d-acaf-4ce87bd484f0-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "97758987-621f-4e7d-acaf-4ce87bd484f0" (UID: "97758987-621f-4e7d-acaf-4ce87bd484f0"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:27:51.925673 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.925619 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97758987-621f-4e7d-acaf-4ce87bd484f0-config-out" (OuterVolumeSpecName: "config-out") pod "97758987-621f-4e7d-acaf-4ce87bd484f0" (UID: "97758987-621f-4e7d-acaf-4ce87bd484f0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:27:51.925800 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.925781 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97758987-621f-4e7d-acaf-4ce87bd484f0-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "97758987-621f-4e7d-acaf-4ce87bd484f0" (UID: "97758987-621f-4e7d-acaf-4ce87bd484f0"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:27:51.925915 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.925878 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "97758987-621f-4e7d-acaf-4ce87bd484f0" (UID: "97758987-621f-4e7d-acaf-4ce87bd484f0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:27:51.926024 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.925918 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97758987-621f-4e7d-acaf-4ce87bd484f0-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "97758987-621f-4e7d-acaf-4ce87bd484f0" (UID: "97758987-621f-4e7d-acaf-4ce87bd484f0"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:27:51.926363 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.926313 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97758987-621f-4e7d-acaf-4ce87bd484f0-kube-api-access-gc2mz" (OuterVolumeSpecName: "kube-api-access-gc2mz") pod "97758987-621f-4e7d-acaf-4ce87bd484f0" (UID: "97758987-621f-4e7d-acaf-4ce87bd484f0"). InnerVolumeSpecName "kube-api-access-gc2mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:27:51.926510 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.926460 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97758987-621f-4e7d-acaf-4ce87bd484f0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "97758987-621f-4e7d-acaf-4ce87bd484f0" (UID: "97758987-621f-4e7d-acaf-4ce87bd484f0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:27:51.926618 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.926497 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "97758987-621f-4e7d-acaf-4ce87bd484f0" (UID: "97758987-621f-4e7d-acaf-4ce87bd484f0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:27:51.926784 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.926757 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "97758987-621f-4e7d-acaf-4ce87bd484f0" (UID: "97758987-621f-4e7d-acaf-4ce87bd484f0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:27:51.928001 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.927978 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-config-volume" (OuterVolumeSpecName: "config-volume") pod "97758987-621f-4e7d-acaf-4ce87bd484f0" (UID: "97758987-621f-4e7d-acaf-4ce87bd484f0"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:27:51.928209 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.928179 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "97758987-621f-4e7d-acaf-4ce87bd484f0" (UID: "97758987-621f-4e7d-acaf-4ce87bd484f0"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:27:51.930029 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.930006 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "97758987-621f-4e7d-acaf-4ce87bd484f0" (UID: "97758987-621f-4e7d-acaf-4ce87bd484f0"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:27:51.935912 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:51.935892 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-web-config" (OuterVolumeSpecName: "web-config") pod "97758987-621f-4e7d-acaf-4ce87bd484f0" (UID: "97758987-621f-4e7d-acaf-4ce87bd484f0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:27:52.024085 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.024060 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-web-config\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:27:52.024085 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.024080 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97758987-621f-4e7d-acaf-4ce87bd484f0-config-out\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:27:52.024085 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.024088 2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-cluster-tls-config\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:27:52.024261 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.024098 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97758987-621f-4e7d-acaf-4ce87bd484f0-metrics-client-ca\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:27:52.024261 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.024106 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc2mz\" (UniqueName: \"kubernetes.io/projected/97758987-621f-4e7d-acaf-4ce87bd484f0-kube-api-access-gc2mz\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:27:52.024261 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.024115 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97758987-621f-4e7d-acaf-4ce87bd484f0-alertmanager-main-db\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:27:52.024261 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.024124 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:27:52.024261 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.024133 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97758987-621f-4e7d-acaf-4ce87bd484f0-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:27:52.024261 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.024142 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-main-tls\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:27:52.024261 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.024151 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:27:52.024261 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.024159 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:27:52.024261 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.024168 2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97758987-621f-4e7d-acaf-4ce87bd484f0-config-volume\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:27:52.024261 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.024176 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97758987-621f-4e7d-acaf-4ce87bd484f0-tls-assets\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:27:52.381334 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.381302 2564 generic.go:356] "Generic (PLEG): container finished" podID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerID="e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f" exitCode=0 Jul 15 19:27:52.381334 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.381325 2564 generic.go:356] "Generic (PLEG): container finished" podID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerID="91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871" exitCode=0 Jul 15 19:27:52.381568 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.381353 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97758987-621f-4e7d-acaf-4ce87bd484f0","Type":"ContainerDied","Data":"e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f"} Jul 15 19:27:52.381568 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.381382 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.381568 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.381409 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97758987-621f-4e7d-acaf-4ce87bd484f0","Type":"ContainerDied","Data":"91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871"} Jul 15 19:27:52.381568 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.381437 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97758987-621f-4e7d-acaf-4ce87bd484f0","Type":"ContainerDied","Data":"6a0dcb3438b4f139ec9b3532cdb7ba501a6f57aa677f76646a746d110923280b"} Jul 15 19:27:52.381568 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.381484 2564 scope.go:117] "RemoveContainer" containerID="8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a" Jul 15 19:27:52.388687 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.388670 2564 scope.go:117] "RemoveContainer" containerID="e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f" Jul 15 19:27:52.395187 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.395171 2564 scope.go:117] "RemoveContainer" containerID="36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743" Jul 15 19:27:52.400178 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.400157 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:27:52.401744 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.401725 2564 scope.go:117] "RemoveContainer" containerID="91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871" Jul 15 19:27:52.406870 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.406850 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:27:52.408373 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.408359 2564 scope.go:117] "RemoveContainer" containerID="f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a" Jul 15 19:27:52.414680 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.414662 2564 scope.go:117] "RemoveContainer" containerID="8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233" Jul 15 19:27:52.421179 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.421164 2564 scope.go:117] "RemoveContainer" containerID="7509db4546e472910d011180a4dbe47c867495656e90340854de30053941eefd" Jul 15 19:27:52.427647 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.427633 2564 scope.go:117] "RemoveContainer" containerID="8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a" Jul 15 19:27:52.427865 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:27:52.427847 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a\": container with ID starting with 8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a not found: ID does not exist" containerID="8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a" Jul 15 19:27:52.427922 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.427872 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a"} err="failed to get container status \"8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a\": rpc error: code = NotFound desc = could not find container \"8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a\": container with ID starting with 8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a not found: ID does not exist" Jul 15 19:27:52.427922 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.427889 2564 scope.go:117] "RemoveContainer" containerID="e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f" Jul 15 19:27:52.428141 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:27:52.428127 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f\": container with ID starting with e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f not found: ID does not exist" containerID="e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f" Jul 15 19:27:52.428176 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.428145 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f"} err="failed to get container status \"e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f\": rpc error: code = NotFound desc = could not find container \"e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f\": container with ID starting with e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f not found: ID does not exist" Jul 15 19:27:52.428176 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.428158 2564 scope.go:117] "RemoveContainer" containerID="36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743" Jul 15 19:27:52.428357 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:27:52.428332 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743\": container with ID starting with 36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743 not found: ID does not exist" containerID="36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743" Jul 15 19:27:52.428389 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.428361 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743"} err="failed to get container status \"36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743\": rpc error: code = NotFound desc = could not find container \"36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743\": container with ID starting with 36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743 not found: ID does not exist" Jul 15 19:27:52.428389 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.428372 2564 scope.go:117] "RemoveContainer" containerID="91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871" Jul 15 19:27:52.428577 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:27:52.428559 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871\": container with ID starting with 91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871 not found: ID does not exist" containerID="91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871" Jul 15 19:27:52.428612 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.428583 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871"} err="failed to get container status \"91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871\": rpc error: code = NotFound desc = could not find container \"91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871\": container with ID starting with 91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871 not found: ID does not exist" Jul 15 19:27:52.428612 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.428596 2564 scope.go:117] "RemoveContainer" containerID="f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a" Jul 15 19:27:52.428808 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:27:52.428790 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a\": container with ID starting with f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a not found: ID does not exist" containerID="f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a" Jul 15 19:27:52.428839 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.428812 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a"} err="failed to get container status \"f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a\": rpc error: code = NotFound desc = could not find container \"f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a\": container with ID starting with f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a not found: ID does not exist" Jul 15 19:27:52.428839 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.428823 2564 scope.go:117] "RemoveContainer" containerID="8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233" Jul 15 19:27:52.429005 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:27:52.428992 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233\": container with ID starting with 8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233 not found: ID does not exist" containerID="8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233" Jul 15 19:27:52.429040 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.429008 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233"} err="failed to get container status \"8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233\": rpc error: code = NotFound desc = could not find container \"8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233\": container with ID starting with 8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233 not found: ID does not exist" Jul 15 19:27:52.429040 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.429019 2564 scope.go:117] "RemoveContainer" containerID="7509db4546e472910d011180a4dbe47c867495656e90340854de30053941eefd" Jul 15 19:27:52.429244 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:27:52.429227 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7509db4546e472910d011180a4dbe47c867495656e90340854de30053941eefd\": container with ID starting with 7509db4546e472910d011180a4dbe47c867495656e90340854de30053941eefd not found: ID does not exist" containerID="7509db4546e472910d011180a4dbe47c867495656e90340854de30053941eefd" Jul 15 19:27:52.429307 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.429253 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7509db4546e472910d011180a4dbe47c867495656e90340854de30053941eefd"} err="failed to get container status \"7509db4546e472910d011180a4dbe47c867495656e90340854de30053941eefd\": rpc error: code = NotFound desc = could not find container \"7509db4546e472910d011180a4dbe47c867495656e90340854de30053941eefd\": container with ID starting with 7509db4546e472910d011180a4dbe47c867495656e90340854de30053941eefd not found: ID does not exist" Jul 15 19:27:52.429307 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.429274 2564 scope.go:117] "RemoveContainer" containerID="8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a" Jul 15 19:27:52.429560 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.429539 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a"} err="failed to get container status \"8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a\": rpc error: code = NotFound desc = could not find container \"8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a\": container with ID starting with 8982362f80156a53ae49892b10537b672c0e664f5cc617cddd288d1e6028e89a not found: ID does not exist" Jul 15 19:27:52.429605 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.429562 2564 scope.go:117] "RemoveContainer" containerID="e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f" Jul 15 19:27:52.429791 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.429774 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f"} err="failed to get container status \"e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f\": rpc error: code = NotFound desc = could not find container \"e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f\": container with ID starting with e3e08ce96f30604eef3674d3c102ea9ef052c966157d09027e9beb5a03608d2f not found: ID does not exist" Jul 15 19:27:52.429835 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.429791 2564 scope.go:117] "RemoveContainer" containerID="36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743" Jul 15 19:27:52.429996 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.429979 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743"} err="failed to get container status \"36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743\": rpc error: code = NotFound desc = could not find container \"36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743\": container with ID starting with 36f79382aa9e5505d0a59e262f6b36cd931fb11ecd6014493d2f7fb75b2c6743 not found: ID does not exist" Jul 15 19:27:52.430061 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.429997 2564 scope.go:117] "RemoveContainer" containerID="91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871" Jul 15 19:27:52.430236 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.430215 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871"} err="failed to get container status \"91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871\": rpc error: code = NotFound desc = could not find container \"91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871\": container with ID starting with 91c947537e0ab8c0379d8bd23e3bb3efc0dd6746531a560601aee85c40f43871 not found: ID does not exist" Jul 15 19:27:52.430298 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.430238 2564 scope.go:117] "RemoveContainer" containerID="f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a" Jul 15 19:27:52.430485 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.430445 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a"} err="failed to get container status \"f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a\": rpc error: code = NotFound desc = could not find container \"f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a\": container with ID starting with f6304e0d969e2ca83d7f4dc57d036b8eb9d4eb09de28071b50426c4dbce5cd8a not found: ID does not exist" Jul 15 19:27:52.430485 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.430463 2564 scope.go:117] "RemoveContainer" containerID="8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233" Jul 15 19:27:52.430677 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.430663 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233"} err="failed to get container status \"8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233\": rpc error: code = NotFound desc = could not find container \"8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233\": container with ID starting with 8f81c02e633c1a8e0704d8444ba9206da8110f59273a6710ad218183d5bfc233 not found: ID does not exist" Jul 15 19:27:52.430740 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.430677 2564 scope.go:117] "RemoveContainer" containerID="7509db4546e472910d011180a4dbe47c867495656e90340854de30053941eefd" Jul 15 19:27:52.430883 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.430863 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7509db4546e472910d011180a4dbe47c867495656e90340854de30053941eefd"} err="failed to get container status \"7509db4546e472910d011180a4dbe47c867495656e90340854de30053941eefd\": rpc error: code = NotFound desc = could not find container \"7509db4546e472910d011180a4dbe47c867495656e90340854de30053941eefd\": container with ID starting with 7509db4546e472910d011180a4dbe47c867495656e90340854de30053941eefd not found: ID does not exist" Jul 15 19:27:52.432518 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432492 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:27:52.432725 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432712 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="alertmanager" Jul 15 19:27:52.432765 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432727 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="alertmanager" Jul 15 19:27:52.432765 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432735 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="kube-rbac-proxy" Jul 15 19:27:52.432765 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432741 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="kube-rbac-proxy" Jul 15 19:27:52.432765 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432747 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="prom-label-proxy" Jul 15 19:27:52.432765 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432752 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="prom-label-proxy" Jul 15 19:27:52.432765 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432760 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="init-config-reloader" Jul 15 19:27:52.432765 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432765 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="init-config-reloader" Jul 15 19:27:52.432955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432773 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="kube-rbac-proxy-web" Jul 15 19:27:52.432955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432778 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="kube-rbac-proxy-web" Jul 15 19:27:52.432955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432784 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="config-reloader" Jul 15 19:27:52.432955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432790 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="config-reloader" Jul 15 19:27:52.432955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432795 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="kube-rbac-proxy-metric" Jul 15 19:27:52.432955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432800 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="kube-rbac-proxy-metric" Jul 15 19:27:52.432955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432842 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="alertmanager" Jul 15 19:27:52.432955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432850 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="config-reloader" Jul 15 19:27:52.432955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432856 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="kube-rbac-proxy" Jul 15 19:27:52.432955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432861 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="kube-rbac-proxy-metric" Jul 15 19:27:52.432955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432866 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="prom-label-proxy" Jul 15 19:27:52.432955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.432871 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" containerName="kube-rbac-proxy-web" Jul 15 19:27:52.437412 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.437396 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.439674 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.439656 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 19:27:52.439773 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.439686 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 19:27:52.439773 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.439694 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 19:27:52.439773 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.439714 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 19:27:52.439907 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.439714 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 19:27:52.439907 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.439838 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8gs8f\"" Jul 15 19:27:52.440098 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.440076 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 19:27:52.440145 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.440101 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 19:27:52.440145 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.440111 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 19:27:52.445068 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.445052 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 19:27:52.449091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.449073 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:27:52.528522 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.528488 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-config-out\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.528522 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.528524 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.528761 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.528543 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.528761 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.528570 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pdcn\" (UniqueName: \"kubernetes.io/projected/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-kube-api-access-4pdcn\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.528761 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.528620 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.528761 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.528673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.528761 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.528704 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.528761 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.528743 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.528955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.528770 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-web-config\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.528955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.528805 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-config-volume\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.528955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.528831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.528955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.528891 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.528955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.528914 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.629329 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.629294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-config-out\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.629530 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.629336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.629530 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.629358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.629530 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.629388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdcn\" (UniqueName: \"kubernetes.io/projected/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-kube-api-access-4pdcn\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.629530 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.629422 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.629530 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.629450 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.629530 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.629501 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.629530 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.629532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.629962 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.629557 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-web-config\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.629962 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.629584 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-config-volume\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.629962 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.629609 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.629962 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.629646 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.629962 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.629669 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.630224 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.630204 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.630514 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.630489 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.632462 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.632400 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.632582 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.632464 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-config-out\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.632815 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.632790 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-config-volume\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.632919 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.632866 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.632919 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.632911 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.633148 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.633125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.633219 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.633125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.633433 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.633353 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.633546 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.633529 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.634791 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.634771 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-web-config\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.637712 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.637696 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pdcn\" (UniqueName: \"kubernetes.io/projected/52bcde0a-20d0-49fc-b970-8d707cf2c5dd-kube-api-access-4pdcn\") pod \"alertmanager-main-0\" (UID: \"52bcde0a-20d0-49fc-b970-8d707cf2c5dd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.745911 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.745878 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:27:52.905736 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:52.905709 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:27:52.908160 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:27:52.908135 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bcde0a_20d0_49fc_b970_8d707cf2c5dd.slice/crio-87f12843e714fa7cc21c526801113a4c2b575bd10ca57fe93c8444e47709f4a0 WatchSource:0}: Error finding container 87f12843e714fa7cc21c526801113a4c2b575bd10ca57fe93c8444e47709f4a0: Status 404 returned error can't find the container with id 87f12843e714fa7cc21c526801113a4c2b575bd10ca57fe93c8444e47709f4a0 Jul 15 19:27:53.385577 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:53.385544 2564 generic.go:356] "Generic (PLEG): container finished" podID="52bcde0a-20d0-49fc-b970-8d707cf2c5dd" containerID="4951051dfa792a49bd28b32ac678e65bcc17a64e5066fe2123278e79415e1458" exitCode=0 Jul 15 19:27:53.385748 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:53.385634 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52bcde0a-20d0-49fc-b970-8d707cf2c5dd","Type":"ContainerDied","Data":"4951051dfa792a49bd28b32ac678e65bcc17a64e5066fe2123278e79415e1458"} Jul 15 19:27:53.385748 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:53.385726 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52bcde0a-20d0-49fc-b970-8d707cf2c5dd","Type":"ContainerStarted","Data":"87f12843e714fa7cc21c526801113a4c2b575bd10ca57fe93c8444e47709f4a0"} Jul 15 19:27:54.058915 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:54.058882 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97758987-621f-4e7d-acaf-4ce87bd484f0" path="/var/lib/kubelet/pods/97758987-621f-4e7d-acaf-4ce87bd484f0/volumes" Jul 15 19:27:54.392643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:54.392556 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52bcde0a-20d0-49fc-b970-8d707cf2c5dd","Type":"ContainerStarted","Data":"03c4b3dade240d88e7590559f37f621b09c0f9219856f16813badca0a5b7df9f"} Jul 15 19:27:54.392643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:54.392635 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52bcde0a-20d0-49fc-b970-8d707cf2c5dd","Type":"ContainerStarted","Data":"724c304162a3db3013d47aac64e744c9866bb95da908bc6259399578f7a9c3d9"} Jul 15 19:27:54.392895 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:54.392679 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52bcde0a-20d0-49fc-b970-8d707cf2c5dd","Type":"ContainerStarted","Data":"532a2435f2cf5a75efe8ea5e2812a97fe02078463752e38bf614e8ec1c3f9768"} Jul 15 19:27:54.392895 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:54.392717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52bcde0a-20d0-49fc-b970-8d707cf2c5dd","Type":"ContainerStarted","Data":"5e0ce50e6aa66a3ae947747ad499af9af49f8e7552588e3cadb58a0fdb5487c9"} Jul 15 19:27:54.392895 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:54.392751 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52bcde0a-20d0-49fc-b970-8d707cf2c5dd","Type":"ContainerStarted","Data":"bc43124ba5355b5509291e5fc7ff31adab841915ee7b3004008ad1499ddf10d7"} Jul 15 19:27:54.392895 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:54.392791 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52bcde0a-20d0-49fc-b970-8d707cf2c5dd","Type":"ContainerStarted","Data":"177ebbf71dbb144bacd46c8e2f763e8d5fd6bbc45b06e9faccf5bcf49248e0e8"} Jul 15 19:27:54.419431 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:54.419381 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.400327013 podStartE2EDuration="2.419366237s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.039224ms" isStatefulPod=false podCreationTimestamp="2026-07-15 19:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:27:54.418536026 +0000 UTC m=+382.890011314" watchObservedRunningTime="2026-07-15 19:27:54.419366237 +0000 UTC m=+382.890841526" Jul 15 19:27:58.345630 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:58.345594 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:58.397561 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:58.397536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:59.674516 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:59.674458 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:59.729075 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:27:59.729040 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:09.556911 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:28:09.556874 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:09.657152 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:28:09.657122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:09.757707 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:28:09.757675 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:10.394690 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:28:10.394652 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:10.688846 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:28:10.688773 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:10.743447 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:28:10.743417 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:46.722798 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:28:46.722768 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:46.776645 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:28:46.776615 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:53.914286 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:28:53.914176 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:56.511490 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:28:56.511451 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:56.612701 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:28:56.612657 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:28.765373 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:28.765315 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:35.186709 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:35.186671 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:35.268334 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:35.268301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:35.478808 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:35.478777 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:35.577159 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:35.577130 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:37.685197 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:37.685153 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:37.896916 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:37.896885 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:38.350068 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:38.350022 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:41.569655 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:41.569605 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:41.670966 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:41.670919 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:45.886761 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:45.886735 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:52.783192 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:52.783145 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:52.881540 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:52.881506 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:54.991861 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:54.991824 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxqn5"] Jul 15 19:29:54.994956 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:54.994935 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxqn5" Jul 15 19:29:54.996983 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:54.996963 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-2lh7v\"" Jul 15 19:29:54.997048 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:54.996998 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:29:54.997861 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:54.997845 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 19:29:55.002930 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:55.002909 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxqn5"] Jul 15 19:29:55.012480 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:55.012450 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd709f77-fa25-477c-b555-b3659f0d5555-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cxqn5\" (UID: \"fd709f77-fa25-477c-b555-b3659f0d5555\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxqn5" Jul 15 19:29:55.012570 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:55.012531 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh4rc\" (UniqueName: \"kubernetes.io/projected/fd709f77-fa25-477c-b555-b3659f0d5555-kube-api-access-jh4rc\") pod \"openshift-lws-operator-fd8ccff4c-cxqn5\" (UID: \"fd709f77-fa25-477c-b555-b3659f0d5555\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxqn5" Jul 15 19:29:55.113797 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:55.113757 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh4rc\" (UniqueName: \"kubernetes.io/projected/fd709f77-fa25-477c-b555-b3659f0d5555-kube-api-access-jh4rc\") pod \"openshift-lws-operator-fd8ccff4c-cxqn5\" (UID: \"fd709f77-fa25-477c-b555-b3659f0d5555\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxqn5" Jul 15 19:29:55.113977 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:55.113925 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd709f77-fa25-477c-b555-b3659f0d5555-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cxqn5\" (UID: \"fd709f77-fa25-477c-b555-b3659f0d5555\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxqn5" Jul 15 19:29:55.114279 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:55.114259 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd709f77-fa25-477c-b555-b3659f0d5555-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cxqn5\" (UID: \"fd709f77-fa25-477c-b555-b3659f0d5555\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxqn5" Jul 15 19:29:55.122324 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:55.122299 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh4rc\" (UniqueName: \"kubernetes.io/projected/fd709f77-fa25-477c-b555-b3659f0d5555-kube-api-access-jh4rc\") pod \"openshift-lws-operator-fd8ccff4c-cxqn5\" (UID: \"fd709f77-fa25-477c-b555-b3659f0d5555\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxqn5" Jul 15 19:29:55.303731 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:55.303633 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxqn5" Jul 15 19:29:55.432793 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:55.432769 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxqn5"] Jul 15 19:29:55.435325 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:29:55.435301 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd709f77_fa25_477c_b555_b3659f0d5555.slice/crio-7cc87cbd2eecbc2825ed3d2c4af8eaf370b4abed23544350aa251799866d04de WatchSource:0}: Error finding container 7cc87cbd2eecbc2825ed3d2c4af8eaf370b4abed23544350aa251799866d04de: Status 404 returned error can't find the container with id 7cc87cbd2eecbc2825ed3d2c4af8eaf370b4abed23544350aa251799866d04de Jul 15 19:29:55.788049 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:55.788018 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxqn5" event={"ID":"fd709f77-fa25-477c-b555-b3659f0d5555","Type":"ContainerStarted","Data":"7cc87cbd2eecbc2825ed3d2c4af8eaf370b4abed23544350aa251799866d04de"} Jul 15 19:29:56.041172 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:56.041097 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2rpjs"] Jul 15 19:29:56.044810 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:56.044794 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2rpjs" Jul 15 19:29:56.047126 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:56.047098 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-grgks\"" Jul 15 19:29:56.047261 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:56.047178 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 19:29:56.047261 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:56.047190 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 19:29:56.063247 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:56.063214 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2rpjs"] Jul 15 19:29:56.122081 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:56.122044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/506b56ec-ab67-4cc0-a561-3915a11e06f0-bound-sa-token\") pod \"cert-manager-55b4d954d5-2rpjs\" (UID: \"506b56ec-ab67-4cc0-a561-3915a11e06f0\") " pod="cert-manager/cert-manager-55b4d954d5-2rpjs" Jul 15 19:29:56.122248 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:56.122091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqcxj\" (UniqueName: \"kubernetes.io/projected/506b56ec-ab67-4cc0-a561-3915a11e06f0-kube-api-access-cqcxj\") pod \"cert-manager-55b4d954d5-2rpjs\" (UID: \"506b56ec-ab67-4cc0-a561-3915a11e06f0\") " pod="cert-manager/cert-manager-55b4d954d5-2rpjs" Jul 15 19:29:56.222461 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:56.222419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/506b56ec-ab67-4cc0-a561-3915a11e06f0-bound-sa-token\") pod \"cert-manager-55b4d954d5-2rpjs\" (UID: \"506b56ec-ab67-4cc0-a561-3915a11e06f0\") " pod="cert-manager/cert-manager-55b4d954d5-2rpjs" Jul 15 19:29:56.222461 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:56.222507 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqcxj\" (UniqueName: \"kubernetes.io/projected/506b56ec-ab67-4cc0-a561-3915a11e06f0-kube-api-access-cqcxj\") pod \"cert-manager-55b4d954d5-2rpjs\" (UID: \"506b56ec-ab67-4cc0-a561-3915a11e06f0\") " pod="cert-manager/cert-manager-55b4d954d5-2rpjs" Jul 15 19:29:56.231189 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:56.231135 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/506b56ec-ab67-4cc0-a561-3915a11e06f0-bound-sa-token\") pod \"cert-manager-55b4d954d5-2rpjs\" (UID: \"506b56ec-ab67-4cc0-a561-3915a11e06f0\") " pod="cert-manager/cert-manager-55b4d954d5-2rpjs" Jul 15 19:29:56.231894 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:56.231867 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqcxj\" (UniqueName: \"kubernetes.io/projected/506b56ec-ab67-4cc0-a561-3915a11e06f0-kube-api-access-cqcxj\") pod \"cert-manager-55b4d954d5-2rpjs\" (UID: \"506b56ec-ab67-4cc0-a561-3915a11e06f0\") " pod="cert-manager/cert-manager-55b4d954d5-2rpjs" Jul 15 19:29:56.353943 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:56.353722 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2rpjs" Jul 15 19:29:56.517885 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:56.517817 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2rpjs"] Jul 15 19:29:56.521995 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:29:56.521778 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod506b56ec_ab67_4cc0_a561_3915a11e06f0.slice/crio-9de5fd02b4d30c2cbd73f28bd7f947cc3b2e2c81d7af20b48773d86a8091867c WatchSource:0}: Error finding container 9de5fd02b4d30c2cbd73f28bd7f947cc3b2e2c81d7af20b48773d86a8091867c: Status 404 returned error can't find the container with id 9de5fd02b4d30c2cbd73f28bd7f947cc3b2e2c81d7af20b48773d86a8091867c Jul 15 19:29:56.793595 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:56.793540 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2rpjs" event={"ID":"506b56ec-ab67-4cc0-a561-3915a11e06f0","Type":"ContainerStarted","Data":"9de5fd02b4d30c2cbd73f28bd7f947cc3b2e2c81d7af20b48773d86a8091867c"} Jul 15 19:29:58.804680 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:58.804639 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxqn5" event={"ID":"fd709f77-fa25-477c-b555-b3659f0d5555","Type":"ContainerStarted","Data":"c74aca5655580be17a4c8bf02b660fcb2e0ba39905e67160d60092cd66c2a018"} Jul 15 19:29:58.821431 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:58.821369 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxqn5" podStartSLOduration=2.209236337 podStartE2EDuration="4.821347357s" totalImagesPullingTime="2.61211102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:58.819533394 +0000 UTC m=+507.291008676" watchObservedRunningTime="2026-07-15 19:29:58.821347357 +0000 UTC m=+507.292822647" Jul 15 19:29:59.659719 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:29:59.659675 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:00.715370 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:00.715321 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:00.813695 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:00.813608 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2rpjs" event={"ID":"506b56ec-ab67-4cc0-a561-3915a11e06f0","Type":"ContainerStarted","Data":"6a80b89cccfab509d8a29d8433ac20195ed6cd78605cb850f96b3ea74873a8b2"} Jul 15 19:30:00.829972 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:00.829923 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-2rpjs" podStartSLOduration=0.803630714 podStartE2EDuration="4.829907188s" totalImagesPullingTime="4.026276474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:29:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:00.828940891 +0000 UTC m=+509.300416181" watchObservedRunningTime="2026-07-15 19:30:00.829907188 +0000 UTC m=+509.301382477" Jul 15 19:30:02.003858 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:02.003831 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:02.063595 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:02.063563 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:02.497090 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:02.497035 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:06.710013 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:06.709979 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:06.794198 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:06.794170 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:20.580785 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:20.580708 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:20.695804 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:20.695746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:21.195129 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.195093 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:21.363222 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.363188 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:21.466653 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.466619 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:21.487302 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.487272 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl"] Jul 15 19:30:21.493624 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.493608 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:21.496614 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.496499 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 19:30:21.496779 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.496764 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 19:30:21.496839 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.496787 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 19:30:21.496877 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.496837 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k7zdf\"" Jul 15 19:30:21.496877 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.496862 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 19:30:21.503627 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.503600 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl"] Jul 15 19:30:21.608740 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.608704 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70c52184-a5a3-4c81-9d01-35650b2fb7b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-df875cfdf-jrzwl\" (UID: \"70c52184-a5a3-4c81-9d01-35650b2fb7b0\") " pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:21.609172 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.608747 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbqhn\" (UniqueName: \"kubernetes.io/projected/70c52184-a5a3-4c81-9d01-35650b2fb7b0-kube-api-access-cbqhn\") pod \"opendatahub-operator-controller-manager-df875cfdf-jrzwl\" (UID: \"70c52184-a5a3-4c81-9d01-35650b2fb7b0\") " pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:21.609172 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.608769 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70c52184-a5a3-4c81-9d01-35650b2fb7b0-tmp\") pod \"opendatahub-operator-controller-manager-df875cfdf-jrzwl\" (UID: \"70c52184-a5a3-4c81-9d01-35650b2fb7b0\") " pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:21.609172 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.608855 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/70c52184-a5a3-4c81-9d01-35650b2fb7b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df875cfdf-jrzwl\" (UID: \"70c52184-a5a3-4c81-9d01-35650b2fb7b0\") " pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:21.609172 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.608898 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/70c52184-a5a3-4c81-9d01-35650b2fb7b0-manifests\") pod \"opendatahub-operator-controller-manager-df875cfdf-jrzwl\" (UID: \"70c52184-a5a3-4c81-9d01-35650b2fb7b0\") " pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:21.692439 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.692399 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:21.710121 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.710092 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70c52184-a5a3-4c81-9d01-35650b2fb7b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-df875cfdf-jrzwl\" (UID: \"70c52184-a5a3-4c81-9d01-35650b2fb7b0\") " pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:21.710236 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.710137 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqhn\" (UniqueName: \"kubernetes.io/projected/70c52184-a5a3-4c81-9d01-35650b2fb7b0-kube-api-access-cbqhn\") pod \"opendatahub-operator-controller-manager-df875cfdf-jrzwl\" (UID: \"70c52184-a5a3-4c81-9d01-35650b2fb7b0\") " pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:21.710297 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.710248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70c52184-a5a3-4c81-9d01-35650b2fb7b0-tmp\") pod \"opendatahub-operator-controller-manager-df875cfdf-jrzwl\" (UID: \"70c52184-a5a3-4c81-9d01-35650b2fb7b0\") " pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:21.710297 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.710281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/70c52184-a5a3-4c81-9d01-35650b2fb7b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df875cfdf-jrzwl\" (UID: \"70c52184-a5a3-4c81-9d01-35650b2fb7b0\") " pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:21.710393 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.710303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/70c52184-a5a3-4c81-9d01-35650b2fb7b0-manifests\") pod \"opendatahub-operator-controller-manager-df875cfdf-jrzwl\" (UID: \"70c52184-a5a3-4c81-9d01-35650b2fb7b0\") " pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:21.710698 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.710665 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/70c52184-a5a3-4c81-9d01-35650b2fb7b0-manifests\") pod \"opendatahub-operator-controller-manager-df875cfdf-jrzwl\" (UID: \"70c52184-a5a3-4c81-9d01-35650b2fb7b0\") " pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:21.712661 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.712634 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70c52184-a5a3-4c81-9d01-35650b2fb7b0-tmp\") pod \"opendatahub-operator-controller-manager-df875cfdf-jrzwl\" (UID: \"70c52184-a5a3-4c81-9d01-35650b2fb7b0\") " pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:21.712767 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.712661 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70c52184-a5a3-4c81-9d01-35650b2fb7b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-df875cfdf-jrzwl\" (UID: \"70c52184-a5a3-4c81-9d01-35650b2fb7b0\") " pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:21.712826 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.712786 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/70c52184-a5a3-4c81-9d01-35650b2fb7b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df875cfdf-jrzwl\" (UID: \"70c52184-a5a3-4c81-9d01-35650b2fb7b0\") " pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:21.719132 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.719080 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbqhn\" (UniqueName: \"kubernetes.io/projected/70c52184-a5a3-4c81-9d01-35650b2fb7b0-kube-api-access-cbqhn\") pod \"opendatahub-operator-controller-manager-df875cfdf-jrzwl\" (UID: \"70c52184-a5a3-4c81-9d01-35650b2fb7b0\") " pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:21.790874 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.790838 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:21.804829 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.804808 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:21.938407 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.938375 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl"] Jul 15 19:30:21.942377 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:30:21.942350 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c52184_a5a3_4c81_9d01_35650b2fb7b0.slice/crio-5486bef9f23d7b0d211f7c54af4e6de163f49092362bafba85749ffbde21f00d WatchSource:0}: Error finding container 5486bef9f23d7b0d211f7c54af4e6de163f49092362bafba85749ffbde21f00d: Status 404 returned error can't find the container with id 5486bef9f23d7b0d211f7c54af4e6de163f49092362bafba85749ffbde21f00d Jul 15 19:30:21.980733 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:21.980652 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:22.046784 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:22.046746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:22.151137 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:22.151103 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:22.248996 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:22.248899 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:22.894765 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:22.894722 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" event={"ID":"70c52184-a5a3-4c81-9d01-35650b2fb7b0","Type":"ContainerStarted","Data":"5486bef9f23d7b0d211f7c54af4e6de163f49092362bafba85749ffbde21f00d"} Jul 15 19:30:25.907646 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:25.907609 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" event={"ID":"70c52184-a5a3-4c81-9d01-35650b2fb7b0","Type":"ContainerStarted","Data":"c078bc9627203516ba911012a7c647550f950946285043bb271edc8518cfb447"} Jul 15 19:30:26.912567 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:26.912529 2564 generic.go:356] "Generic (PLEG): container finished" podID="70c52184-a5a3-4c81-9d01-35650b2fb7b0" containerID="c078bc9627203516ba911012a7c647550f950946285043bb271edc8518cfb447" exitCode=0 Jul 15 19:30:26.912955 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:26.912606 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" event={"ID":"70c52184-a5a3-4c81-9d01-35650b2fb7b0","Type":"ContainerDied","Data":"c078bc9627203516ba911012a7c647550f950946285043bb271edc8518cfb447"} Jul 15 19:30:27.919089 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:27.919050 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" event={"ID":"70c52184-a5a3-4c81-9d01-35650b2fb7b0","Type":"ContainerStarted","Data":"2c4532c0f373bb4e7d5d90fdcaee5f1ad2a9e24578fa87ace345d11858596202"} Jul 15 19:30:27.919457 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:27.919134 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:27.941531 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:27.941447 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" podStartSLOduration=2.338203703 podStartE2EDuration="6.941431883s" totalImagesPullingTime="3.592488723s" totalInitContainerRuntime="1.010739457s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:30:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:27.940493265 +0000 UTC m=+536.411968553" watchObservedRunningTime="2026-07-15 19:30:27.941431883 +0000 UTC m=+536.412907172" Jul 15 19:30:32.769125 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:32.769090 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:32.864683 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:32.864650 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:34.466496 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:34.466448 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:34.561394 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:34.561362 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:38.924785 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:38.924757 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-df875cfdf-jrzwl" Jul 15 19:30:41.548861 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:41.548818 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:41.743877 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:41.743838 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:41.911645 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:41.911560 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:41.961304 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:41.961272 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6d4465d888-r29kn"] Jul 15 19:30:41.967390 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:41.967364 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d4465d888-r29kn" Jul 15 19:30:41.975998 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:41.975972 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 19:30:41.976134 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:41.976007 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 19:30:41.976134 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:41.976128 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 19:30:41.976442 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:41.976421 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d4465d888-r29kn"] Jul 15 19:30:41.976549 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:41.976428 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 19:30:42.072421 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.072395 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmpsq\" (UniqueName: \"kubernetes.io/projected/3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3-kube-api-access-wmpsq\") pod \"kube-auth-proxy-6d4465d888-r29kn\" (UID: \"3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3\") " pod="openshift-ingress/kube-auth-proxy-6d4465d888-r29kn" Jul 15 19:30:42.072602 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.072440 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3-tls-certs\") pod \"kube-auth-proxy-6d4465d888-r29kn\" (UID: \"3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3\") " pod="openshift-ingress/kube-auth-proxy-6d4465d888-r29kn" Jul 15 19:30:42.072602 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.072515 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3-tmp\") pod \"kube-auth-proxy-6d4465d888-r29kn\" (UID: \"3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3\") " pod="openshift-ingress/kube-auth-proxy-6d4465d888-r29kn" Jul 15 19:30:42.140548 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.140512 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:42.173747 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.173651 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3-tls-certs\") pod \"kube-auth-proxy-6d4465d888-r29kn\" (UID: \"3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3\") " pod="openshift-ingress/kube-auth-proxy-6d4465d888-r29kn" Jul 15 19:30:42.173747 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.173706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3-tmp\") pod \"kube-auth-proxy-6d4465d888-r29kn\" (UID: \"3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3\") " pod="openshift-ingress/kube-auth-proxy-6d4465d888-r29kn" Jul 15 19:30:42.173747 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.173739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmpsq\" (UniqueName: \"kubernetes.io/projected/3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3-kube-api-access-wmpsq\") pod \"kube-auth-proxy-6d4465d888-r29kn\" (UID: \"3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3\") " pod="openshift-ingress/kube-auth-proxy-6d4465d888-r29kn" Jul 15 19:30:42.176289 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.176249 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3-tmp\") pod \"kube-auth-proxy-6d4465d888-r29kn\" (UID: \"3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3\") " pod="openshift-ingress/kube-auth-proxy-6d4465d888-r29kn" Jul 15 19:30:42.176494 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.176455 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3-tls-certs\") pod \"kube-auth-proxy-6d4465d888-r29kn\" (UID: \"3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3\") " pod="openshift-ingress/kube-auth-proxy-6d4465d888-r29kn" Jul 15 19:30:42.182391 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.182359 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmpsq\" (UniqueName: \"kubernetes.io/projected/3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3-kube-api-access-wmpsq\") pod \"kube-auth-proxy-6d4465d888-r29kn\" (UID: \"3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3\") " pod="openshift-ingress/kube-auth-proxy-6d4465d888-r29kn" Jul 15 19:30:42.278047 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.278005 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d4465d888-r29kn" Jul 15 19:30:42.280183 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.280154 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qtll6"] Jul 15 19:30:42.282450 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.282435 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qtll6" Jul 15 19:30:42.285852 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.285835 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-89w9s\"" Jul 15 19:30:42.285958 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.285839 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 15 19:30:42.304609 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.304580 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qtll6"] Jul 15 19:30:42.336486 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.336411 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:42.376106 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.376081 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/594a348b-5c74-4c66-9f9b-f94cfacf9fb6-cert\") pod \"odh-model-controller-7998c67dc4-qtll6\" (UID: \"594a348b-5c74-4c66-9f9b-f94cfacf9fb6\") " pod="opendatahub/odh-model-controller-7998c67dc4-qtll6" Jul 15 19:30:42.376239 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.376164 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgbnw\" (UniqueName: \"kubernetes.io/projected/594a348b-5c74-4c66-9f9b-f94cfacf9fb6-kube-api-access-kgbnw\") pod \"odh-model-controller-7998c67dc4-qtll6\" (UID: \"594a348b-5c74-4c66-9f9b-f94cfacf9fb6\") " pod="opendatahub/odh-model-controller-7998c67dc4-qtll6" Jul 15 19:30:42.429347 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.429324 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d4465d888-r29kn"] Jul 15 19:30:42.431636 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:30:42.431609 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd1b2ec_4ec4_4026_b4c7_c3dab455b9e3.slice/crio-40470e55a63219a21e59dd7158c7e741e62d720a9726d9829f85c7860dc06c8b WatchSource:0}: Error finding container 40470e55a63219a21e59dd7158c7e741e62d720a9726d9829f85c7860dc06c8b: Status 404 returned error can't find the container with id 40470e55a63219a21e59dd7158c7e741e62d720a9726d9829f85c7860dc06c8b Jul 15 19:30:42.477230 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.477189 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbnw\" (UniqueName: \"kubernetes.io/projected/594a348b-5c74-4c66-9f9b-f94cfacf9fb6-kube-api-access-kgbnw\") pod \"odh-model-controller-7998c67dc4-qtll6\" (UID: \"594a348b-5c74-4c66-9f9b-f94cfacf9fb6\") " pod="opendatahub/odh-model-controller-7998c67dc4-qtll6" Jul 15 19:30:42.477417 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.477264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/594a348b-5c74-4c66-9f9b-f94cfacf9fb6-cert\") pod \"odh-model-controller-7998c67dc4-qtll6\" (UID: \"594a348b-5c74-4c66-9f9b-f94cfacf9fb6\") " pod="opendatahub/odh-model-controller-7998c67dc4-qtll6" Jul 15 19:30:42.477417 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:30:42.477349 2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 15 19:30:42.477417 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:30:42.477416 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/594a348b-5c74-4c66-9f9b-f94cfacf9fb6-cert podName:594a348b-5c74-4c66-9f9b-f94cfacf9fb6 nodeName:}" failed. No retries permitted until 2026-07-15 19:30:42.977394756 +0000 UTC m=+551.448870023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/594a348b-5c74-4c66-9f9b-f94cfacf9fb6-cert") pod "odh-model-controller-7998c67dc4-qtll6" (UID: "594a348b-5c74-4c66-9f9b-f94cfacf9fb6") : secret "odh-model-controller-webhook-cert" not found Jul 15 19:30:42.487837 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.487809 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbnw\" (UniqueName: \"kubernetes.io/projected/594a348b-5c74-4c66-9f9b-f94cfacf9fb6-kube-api-access-kgbnw\") pod \"odh-model-controller-7998c67dc4-qtll6\" (UID: \"594a348b-5c74-4c66-9f9b-f94cfacf9fb6\") " pod="opendatahub/odh-model-controller-7998c67dc4-qtll6" Jul 15 19:30:42.979738 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.979702 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d4465d888-r29kn" event={"ID":"3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3","Type":"ContainerStarted","Data":"40470e55a63219a21e59dd7158c7e741e62d720a9726d9829f85c7860dc06c8b"} Jul 15 19:30:42.982083 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.982054 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/594a348b-5c74-4c66-9f9b-f94cfacf9fb6-cert\") pod \"odh-model-controller-7998c67dc4-qtll6\" (UID: \"594a348b-5c74-4c66-9f9b-f94cfacf9fb6\") " pod="opendatahub/odh-model-controller-7998c67dc4-qtll6" Jul 15 19:30:42.984586 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:42.984563 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/594a348b-5c74-4c66-9f9b-f94cfacf9fb6-cert\") pod \"odh-model-controller-7998c67dc4-qtll6\" (UID: \"594a348b-5c74-4c66-9f9b-f94cfacf9fb6\") " pod="opendatahub/odh-model-controller-7998c67dc4-qtll6" Jul 15 19:30:43.145027 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.144993 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:43.207136 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.207104 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qtll6" Jul 15 19:30:43.251004 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.250920 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:43.336245 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.336198 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:43.343053 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.343022 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vxlh2"] Jul 15 19:30:43.345819 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.345801 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vxlh2" Jul 15 19:30:43.349348 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.349324 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 15 19:30:43.349494 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.349332 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-gn8fm\"" Jul 15 19:30:43.352883 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.352619 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qtll6"] Jul 15 19:30:43.357071 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:30:43.357041 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594a348b_5c74_4c66_9f9b_f94cfacf9fb6.slice/crio-e5208565b347c493fcc3738dd45b989603b5e99641eb1e17423080f2861d83e0 WatchSource:0}: Error finding container e5208565b347c493fcc3738dd45b989603b5e99641eb1e17423080f2861d83e0: Status 404 returned error can't find the container with id e5208565b347c493fcc3738dd45b989603b5e99641eb1e17423080f2861d83e0 Jul 15 19:30:43.359706 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.359682 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vxlh2"] Jul 15 19:30:43.413434 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.413399 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:43.486819 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.486779 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dclw\" (UniqueName: \"kubernetes.io/projected/d0413cf3-332b-4ea7-8ca2-a90586a494fc-kube-api-access-2dclw\") pod \"model-serving-api-54d89bd5fc-vxlh2\" (UID: \"d0413cf3-332b-4ea7-8ca2-a90586a494fc\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxlh2" Jul 15 19:30:43.487005 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.486836 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0413cf3-332b-4ea7-8ca2-a90586a494fc-tls-certs\") pod \"model-serving-api-54d89bd5fc-vxlh2\" (UID: \"d0413cf3-332b-4ea7-8ca2-a90586a494fc\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxlh2" Jul 15 19:30:43.581953 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.581868 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8"] Jul 15 19:30:43.584183 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.584154 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.584400 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.584374 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:43.586323 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.586306 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 15 19:30:43.586501 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.586460 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 19:30:43.586869 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.586851 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 15 19:30:43.586984 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.586868 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 15 19:30:43.586984 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.586886 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qxj9l\"" Jul 15 19:30:43.587252 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.587229 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dclw\" (UniqueName: \"kubernetes.io/projected/d0413cf3-332b-4ea7-8ca2-a90586a494fc-kube-api-access-2dclw\") pod \"model-serving-api-54d89bd5fc-vxlh2\" (UID: \"d0413cf3-332b-4ea7-8ca2-a90586a494fc\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxlh2" Jul 15 19:30:43.587311 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.587257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0413cf3-332b-4ea7-8ca2-a90586a494fc-tls-certs\") pod \"model-serving-api-54d89bd5fc-vxlh2\" (UID: \"d0413cf3-332b-4ea7-8ca2-a90586a494fc\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxlh2" Jul 15 19:30:43.590120 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.590097 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0413cf3-332b-4ea7-8ca2-a90586a494fc-tls-certs\") pod \"model-serving-api-54d89bd5fc-vxlh2\" (UID: \"d0413cf3-332b-4ea7-8ca2-a90586a494fc\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxlh2" Jul 15 19:30:43.598302 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.598270 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8"] Jul 15 19:30:43.601898 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.601868 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dclw\" (UniqueName: \"kubernetes.io/projected/d0413cf3-332b-4ea7-8ca2-a90586a494fc-kube-api-access-2dclw\") pod \"model-serving-api-54d89bd5fc-vxlh2\" (UID: \"d0413cf3-332b-4ea7-8ca2-a90586a494fc\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxlh2" Jul 15 19:30:43.659159 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.659048 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vxlh2" Jul 15 19:30:43.688020 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.687991 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.688177 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.688048 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-token\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.688177 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.688091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.688177 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.688118 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d33c40b-8842-42dd-b3c9-6fa76498d82e-local-certs\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.688325 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.688198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.688325 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.688236 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-cacerts\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.688325 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.688278 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxnzz\" (UniqueName: \"kubernetes.io/projected/3d33c40b-8842-42dd-b3c9-6fa76498d82e-kube-api-access-sxnzz\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.789057 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.789006 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-token\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.789253 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.789063 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.789253 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.789098 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d33c40b-8842-42dd-b3c9-6fa76498d82e-local-certs\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.789253 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.789135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.789253 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.789159 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-cacerts\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.789253 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.789199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxnzz\" (UniqueName: \"kubernetes.io/projected/3d33c40b-8842-42dd-b3c9-6fa76498d82e-kube-api-access-sxnzz\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.789253 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.789244 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.789763 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.789736 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.792732 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.792691 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d33c40b-8842-42dd-b3c9-6fa76498d82e-local-certs\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.792732 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.792726 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.792971 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.792752 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-cacerts\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.792971 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.792767 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.800691 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.800664 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-token\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.800821 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.800665 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxnzz\" (UniqueName: \"kubernetes.io/projected/3d33c40b-8842-42dd-b3c9-6fa76498d82e-kube-api-access-sxnzz\") pod \"istiod-openshift-gateway-7574459656-hjhq8\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.812026 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.811973 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vxlh2"] Jul 15 19:30:43.814864 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:30:43.814833 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0413cf3_332b_4ea7_8ca2_a90586a494fc.slice/crio-742ebef10eaee26d4ddab3dea2b9e6fea900f8f2743ee75f6756aacbb04c7954 WatchSource:0}: Error finding container 742ebef10eaee26d4ddab3dea2b9e6fea900f8f2743ee75f6756aacbb04c7954: Status 404 returned error can't find the container with id 742ebef10eaee26d4ddab3dea2b9e6fea900f8f2743ee75f6756aacbb04c7954 Jul 15 19:30:43.815439 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.815410 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:43.903787 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.903702 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:43.984843 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.984810 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vxlh2" event={"ID":"d0413cf3-332b-4ea7-8ca2-a90586a494fc","Type":"ContainerStarted","Data":"742ebef10eaee26d4ddab3dea2b9e6fea900f8f2743ee75f6756aacbb04c7954"} Jul 15 19:30:43.986006 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:43.985981 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qtll6" event={"ID":"594a348b-5c74-4c66-9f9b-f94cfacf9fb6","Type":"ContainerStarted","Data":"e5208565b347c493fcc3738dd45b989603b5e99641eb1e17423080f2861d83e0"} Jul 15 19:30:44.050130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:44.050082 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8"] Jul 15 19:30:44.055830 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:30:44.054912 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d33c40b_8842_42dd_b3c9_6fa76498d82e.slice/crio-b30c529b46a1847271cda1481149649995eedcdea263acff7b6cb746d171f436 WatchSource:0}: Error finding container b30c529b46a1847271cda1481149649995eedcdea263acff7b6cb746d171f436: Status 404 returned error can't find the container with id b30c529b46a1847271cda1481149649995eedcdea263acff7b6cb746d171f436 Jul 15 19:30:44.770533 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:44.770500 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:44.998001 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:44.997960 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" event={"ID":"3d33c40b-8842-42dd-b3c9-6fa76498d82e","Type":"ContainerStarted","Data":"b30c529b46a1847271cda1481149649995eedcdea263acff7b6cb746d171f436"} Jul 15 19:30:48.784585 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:48.784548 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 19:30:48.784963 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:48.784628 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 19:30:49.018726 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:49.018675 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vxlh2" event={"ID":"d0413cf3-332b-4ea7-8ca2-a90586a494fc","Type":"ContainerStarted","Data":"26e2b31e19e03f3895f0281488b76dd8f1fc77a09c61dcee547bfbcbf42f903d"} Jul 15 19:30:49.018956 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:49.018915 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-vxlh2" Jul 15 19:30:49.020567 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:49.020500 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d4465d888-r29kn" event={"ID":"3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3","Type":"ContainerStarted","Data":"61598c09fa7842f631d5e49bac6f0f6f1595d0f645934cd613b174e0cbe48e64"} Jul 15 19:30:49.021961 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:49.021937 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qtll6" event={"ID":"594a348b-5c74-4c66-9f9b-f94cfacf9fb6","Type":"ContainerStarted","Data":"e584f111864c86c1513839f91bda4d05936b59263ec3b6c53a011b5be670a142"} Jul 15 19:30:49.022379 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:49.022314 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-qtll6" Jul 15 19:30:49.024043 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:49.024018 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" event={"ID":"3d33c40b-8842-42dd-b3c9-6fa76498d82e","Type":"ContainerStarted","Data":"b6a8ab1f3e2bde3e893b043132911c0b6a1c61380d0b15b3ac1969febb0b5aa5"} Jul 15 19:30:49.024218 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:49.024203 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:49.026059 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:49.026033 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-hjhq8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 19:30:49.026156 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:49.026087 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" podUID="3d33c40b-8842-42dd-b3c9-6fa76498d82e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 19:30:49.036273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:49.036233 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-vxlh2" podStartSLOduration=1.084911182 podStartE2EDuration="6.036220087s" totalImagesPullingTime="4.951308905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:49.034736455 +0000 UTC m=+557.506211742" watchObservedRunningTime="2026-07-15 19:30:49.036220087 +0000 UTC m=+557.507695369" Jul 15 19:30:49.050679 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:49.050627 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6d4465d888-r29kn" podStartSLOduration=1.704933455 podStartE2EDuration="8.050609816s" totalImagesPullingTime="6.345676361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:30:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:49.049368197 +0000 UTC m=+557.520843488" watchObservedRunningTime="2026-07-15 19:30:49.050609816 +0000 UTC m=+557.522085103" Jul 15 19:30:49.064274 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:49.064210 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-qtll6" podStartSLOduration=1.654392917 podStartE2EDuration="7.064199389s" totalImagesPullingTime="5.409806472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:49.063789574 +0000 UTC m=+557.535264864" watchObservedRunningTime="2026-07-15 19:30:49.064199389 +0000 UTC m=+557.535674679" Jul 15 19:30:49.084129 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:49.083962 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" podStartSLOduration=1.358484268 podStartE2EDuration="6.08394361s" totalImagesPullingTime="4.725459342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:49.083556397 +0000 UTC m=+557.555031721" watchObservedRunningTime="2026-07-15 19:30:49.08394361 +0000 UTC m=+557.555418900" Jul 15 19:30:49.795757 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:49.795708 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:50.028931 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.028898 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-hjhq8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 19:30:50.029120 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.028956 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" podUID="3d33c40b-8842-42dd-b3c9-6fa76498d82e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 19:30:50.619193 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.619163 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h"] Jul 15 19:30:50.621588 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.621567 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.624185 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.624161 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-9st7f\"" Jul 15 19:30:50.638408 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.638379 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h"] Jul 15 19:30:50.754684 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.754646 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f8ab2c-946b-4be7-9708-1a6f8041d68c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.754832 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.754706 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f8ab2c-946b-4be7-9708-1a6f8041d68c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.754832 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.754775 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrqkb\" (UniqueName: \"kubernetes.io/projected/07f8ab2c-946b-4be7-9708-1a6f8041d68c-kube-api-access-xrqkb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.754832 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.754823 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07f8ab2c-946b-4be7-9708-1a6f8041d68c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.754949 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.754849 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f8ab2c-946b-4be7-9708-1a6f8041d68c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.754949 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.754865 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f8ab2c-946b-4be7-9708-1a6f8041d68c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.754949 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.754882 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f8ab2c-946b-4be7-9708-1a6f8041d68c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.754949 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.754902 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f8ab2c-946b-4be7-9708-1a6f8041d68c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.754949 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.754931 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f8ab2c-946b-4be7-9708-1a6f8041d68c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.855891 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.855857 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f8ab2c-946b-4be7-9708-1a6f8041d68c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.856337 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.855910 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f8ab2c-946b-4be7-9708-1a6f8041d68c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.856337 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.855977 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f8ab2c-946b-4be7-9708-1a6f8041d68c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.856337 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.856005 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrqkb\" (UniqueName: \"kubernetes.io/projected/07f8ab2c-946b-4be7-9708-1a6f8041d68c-kube-api-access-xrqkb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.856337 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.856052 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07f8ab2c-946b-4be7-9708-1a6f8041d68c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.856337 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.856087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f8ab2c-946b-4be7-9708-1a6f8041d68c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.856337 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.856109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f8ab2c-946b-4be7-9708-1a6f8041d68c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.856337 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.856132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f8ab2c-946b-4be7-9708-1a6f8041d68c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.856337 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.856166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f8ab2c-946b-4be7-9708-1a6f8041d68c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.856337 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.856234 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f8ab2c-946b-4be7-9708-1a6f8041d68c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.856337 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.856277 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f8ab2c-946b-4be7-9708-1a6f8041d68c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.856847 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.856579 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f8ab2c-946b-4be7-9708-1a6f8041d68c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.856847 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.856680 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f8ab2c-946b-4be7-9708-1a6f8041d68c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.856913 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.856895 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f8ab2c-946b-4be7-9708-1a6f8041d68c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.858440 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.858419 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f8ab2c-946b-4be7-9708-1a6f8041d68c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.858561 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.858491 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f8ab2c-946b-4be7-9708-1a6f8041d68c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.865734 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.865713 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrqkb\" (UniqueName: \"kubernetes.io/projected/07f8ab2c-946b-4be7-9708-1a6f8041d68c-kube-api-access-xrqkb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.870920 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.870865 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07f8ab2c-946b-4be7-9708-1a6f8041d68c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4llb5h\" (UID: \"07f8ab2c-946b-4be7-9708-1a6f8041d68c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:50.933067 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:50.933037 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:51.076952 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:51.076924 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h"] Jul 15 19:30:51.080093 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:30:51.080070 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f8ab2c_946b_4be7_9708_1a6f8041d68c.slice/crio-470b77b567a0b5d31ce6e81d063c3c13956456cc6c421a5e5f0fe78528aaf31d WatchSource:0}: Error finding container 470b77b567a0b5d31ce6e81d063c3c13956456cc6c421a5e5f0fe78528aaf31d: Status 404 returned error can't find the container with id 470b77b567a0b5d31ce6e81d063c3c13956456cc6c421a5e5f0fe78528aaf31d Jul 15 19:30:51.923275 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:51.923221 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:52.038499 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:52.038456 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" event={"ID":"07f8ab2c-946b-4be7-9708-1a6f8041d68c","Type":"ContainerStarted","Data":"470b77b567a0b5d31ce6e81d063c3c13956456cc6c421a5e5f0fe78528aaf31d"} Jul 15 19:30:53.029139 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.029108 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:53.748351 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.748317 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:53.830648 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.830614 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nlmpm"] Jul 15 19:30:53.835762 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.835732 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nlmpm" Jul 15 19:30:53.838054 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.838022 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 15 19:30:53.838190 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.838065 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-c44qv\"" Jul 15 19:30:53.854570 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.854531 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nlmpm"] Jul 15 19:30:53.856140 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.856114 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-76jrd"] Jul 15 19:30:53.858522 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.858456 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-76jrd" Jul 15 19:30:53.860795 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.860674 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 15 19:30:53.860795 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.860710 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-d89q9\"" Jul 15 19:30:53.869092 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.869069 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-76jrd"] Jul 15 19:30:53.896860 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.896836 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1430a43-7fd3-4989-ab2d-1047ea045c53-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-76jrd\" (UID: \"c1430a43-7fd3-4989-ab2d-1047ea045c53\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-76jrd" Jul 15 19:30:53.896993 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.896876 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcb8c706-6b2d-4df0-99b8-294b0e3f4221-cert\") pod \"kserve-controller-manager-d6f4b65b8-nlmpm\" (UID: \"dcb8c706-6b2d-4df0-99b8-294b0e3f4221\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nlmpm" Jul 15 19:30:53.896993 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.896973 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rksc8\" (UniqueName: \"kubernetes.io/projected/dcb8c706-6b2d-4df0-99b8-294b0e3f4221-kube-api-access-rksc8\") pod \"kserve-controller-manager-d6f4b65b8-nlmpm\" (UID: \"dcb8c706-6b2d-4df0-99b8-294b0e3f4221\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nlmpm" Jul 15 19:30:53.897096 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.897009 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd6bm\" (UniqueName: \"kubernetes.io/projected/c1430a43-7fd3-4989-ab2d-1047ea045c53-kube-api-access-rd6bm\") pod \"llmisvc-controller-manager-777c6bd7d5-76jrd\" (UID: \"c1430a43-7fd3-4989-ab2d-1047ea045c53\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-76jrd" Jul 15 19:30:53.907355 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.907332 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:53.998245 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.998206 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksc8\" (UniqueName: \"kubernetes.io/projected/dcb8c706-6b2d-4df0-99b8-294b0e3f4221-kube-api-access-rksc8\") pod \"kserve-controller-manager-d6f4b65b8-nlmpm\" (UID: \"dcb8c706-6b2d-4df0-99b8-294b0e3f4221\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nlmpm" Jul 15 19:30:53.998463 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.998257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd6bm\" (UniqueName: \"kubernetes.io/projected/c1430a43-7fd3-4989-ab2d-1047ea045c53-kube-api-access-rd6bm\") pod \"llmisvc-controller-manager-777c6bd7d5-76jrd\" (UID: \"c1430a43-7fd3-4989-ab2d-1047ea045c53\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-76jrd" Jul 15 19:30:53.998463 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.998289 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1430a43-7fd3-4989-ab2d-1047ea045c53-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-76jrd\" (UID: \"c1430a43-7fd3-4989-ab2d-1047ea045c53\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-76jrd" Jul 15 19:30:53.998463 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:53.998326 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcb8c706-6b2d-4df0-99b8-294b0e3f4221-cert\") pod \"kserve-controller-manager-d6f4b65b8-nlmpm\" (UID: \"dcb8c706-6b2d-4df0-99b8-294b0e3f4221\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nlmpm" Jul 15 19:30:53.998463 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:30:53.998423 2564 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 15 19:30:53.998745 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:30:53.998511 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1430a43-7fd3-4989-ab2d-1047ea045c53-cert podName:c1430a43-7fd3-4989-ab2d-1047ea045c53 nodeName:}" failed. No retries permitted until 2026-07-15 19:30:54.498465415 +0000 UTC m=+562.969940686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c1430a43-7fd3-4989-ab2d-1047ea045c53-cert") pod "llmisvc-controller-manager-777c6bd7d5-76jrd" (UID: "c1430a43-7fd3-4989-ab2d-1047ea045c53") : secret "llmisvc-webhook-server-cert" not found Jul 15 19:30:54.001202 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:54.001180 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcb8c706-6b2d-4df0-99b8-294b0e3f4221-cert\") pod \"kserve-controller-manager-d6f4b65b8-nlmpm\" (UID: \"dcb8c706-6b2d-4df0-99b8-294b0e3f4221\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nlmpm" Jul 15 19:30:54.010687 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:54.010653 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:54.012871 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:54.012847 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rksc8\" (UniqueName: \"kubernetes.io/projected/dcb8c706-6b2d-4df0-99b8-294b0e3f4221-kube-api-access-rksc8\") pod \"kserve-controller-manager-d6f4b65b8-nlmpm\" (UID: \"dcb8c706-6b2d-4df0-99b8-294b0e3f4221\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nlmpm" Jul 15 19:30:54.013939 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:54.013917 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd6bm\" (UniqueName: \"kubernetes.io/projected/c1430a43-7fd3-4989-ab2d-1047ea045c53-kube-api-access-rd6bm\") pod \"llmisvc-controller-manager-777c6bd7d5-76jrd\" (UID: \"c1430a43-7fd3-4989-ab2d-1047ea045c53\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-76jrd" Jul 15 19:30:54.148395 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:54.148365 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nlmpm" Jul 15 19:30:54.164131 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:54.164094 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:54.503021 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:54.502987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1430a43-7fd3-4989-ab2d-1047ea045c53-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-76jrd\" (UID: \"c1430a43-7fd3-4989-ab2d-1047ea045c53\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-76jrd" Jul 15 19:30:54.509232 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:54.509174 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1430a43-7fd3-4989-ab2d-1047ea045c53-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-76jrd\" (UID: \"c1430a43-7fd3-4989-ab2d-1047ea045c53\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-76jrd" Jul 15 19:30:54.514058 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:54.514025 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 19:30:54.514321 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:54.514111 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 19:30:54.514321 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:54.514149 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 19:30:54.597425 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:54.597396 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nlmpm"] Jul 15 19:30:54.599309 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:30:54.599271 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb8c706_6b2d_4df0_99b8_294b0e3f4221.slice/crio-80b7c73e89312fbf37ce9cde91c2e23a63991cde732e4f58c884b399fbba2ffb WatchSource:0}: Error finding container 80b7c73e89312fbf37ce9cde91c2e23a63991cde732e4f58c884b399fbba2ffb: Status 404 returned error can't find the container with id 80b7c73e89312fbf37ce9cde91c2e23a63991cde732e4f58c884b399fbba2ffb Jul 15 19:30:54.770945 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:54.770867 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-76jrd" Jul 15 19:30:54.902278 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:54.902246 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-76jrd"] Jul 15 19:30:54.904055 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:30:54.904028 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc1430a43_7fd3_4989_ab2d_1047ea045c53.slice/crio-418e0a3529df757de558d754721ead9a967de8e891a649ed651ba8d9952d4be6 WatchSource:0}: Error finding container 418e0a3529df757de558d754721ead9a967de8e891a649ed651ba8d9952d4be6: Status 404 returned error can't find the container with id 418e0a3529df757de558d754721ead9a967de8e891a649ed651ba8d9952d4be6 Jul 15 19:30:55.053183 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:55.053088 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-76jrd" event={"ID":"c1430a43-7fd3-4989-ab2d-1047ea045c53","Type":"ContainerStarted","Data":"418e0a3529df757de558d754721ead9a967de8e891a649ed651ba8d9952d4be6"} Jul 15 19:30:55.054424 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:55.054393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nlmpm" event={"ID":"dcb8c706-6b2d-4df0-99b8-294b0e3f4221","Type":"ContainerStarted","Data":"80b7c73e89312fbf37ce9cde91c2e23a63991cde732e4f58c884b399fbba2ffb"} Jul 15 19:30:55.055934 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:55.055908 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" event={"ID":"07f8ab2c-946b-4be7-9708-1a6f8041d68c","Type":"ContainerStarted","Data":"48a8afb1d73b38e08186e4015e50cb35e27a0b2ed07340a14d7bb8eb566b2aa4"} Jul 15 19:30:55.062586 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:55.062549 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:55.161935 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:55.161895 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:55.260810 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:55.260782 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:55.933826 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:55.933791 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:55.939101 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:55.939075 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:55.962631 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:55.962569 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" podStartSLOduration=2.530881286 podStartE2EDuration="5.962546246s" totalImagesPullingTime="3.43166496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:55.103738791 +0000 UTC m=+563.575214080" watchObservedRunningTime="2026-07-15 19:30:55.962546246 +0000 UTC m=+564.434021536" Jul 15 19:30:56.059483 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:56.059439 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:56.060339 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:56.060319 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4llb5h" Jul 15 19:30:57.521211 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:57.521173 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8"] Jul 15 19:30:57.521726 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:57.521404 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" podUID="3d33c40b-8842-42dd-b3c9-6fa76498d82e" containerName="discovery" containerID="cri-o://b6a8ab1f3e2bde3e893b043132911c0b6a1c61380d0b15b3ac1969febb0b5aa5" gracePeriod=30 Jul 15 19:30:58.070420 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.070390 2564 generic.go:356] "Generic (PLEG): container finished" podID="3d33c40b-8842-42dd-b3c9-6fa76498d82e" containerID="b6a8ab1f3e2bde3e893b043132911c0b6a1c61380d0b15b3ac1969febb0b5aa5" exitCode=0 Jul 15 19:30:58.070545 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.070439 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" event={"ID":"3d33c40b-8842-42dd-b3c9-6fa76498d82e","Type":"ContainerDied","Data":"b6a8ab1f3e2bde3e893b043132911c0b6a1c61380d0b15b3ac1969febb0b5aa5"} Jul 15 19:30:58.072112 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.072086 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nlmpm" event={"ID":"dcb8c706-6b2d-4df0-99b8-294b0e3f4221","Type":"ContainerStarted","Data":"fe76b71ff1b3b63f7c3ea118fd57e2c6ce80eee47954dde90ff94db271f825bd"} Jul 15 19:30:58.093344 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.093298 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nlmpm" podStartSLOduration=2.260248448 podStartE2EDuration="5.093281453s" totalImagesPullingTime="2.833033005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:58.091767999 +0000 UTC m=+566.563243292" watchObservedRunningTime="2026-07-15 19:30:58.093281453 +0000 UTC m=+566.564756742" Jul 15 19:30:58.182870 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.182844 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:58.238517 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.238485 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-csr-dns-cert\") pod \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " Jul 15 19:30:58.238690 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.238540 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-csr-ca-configmap\") pod \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " Jul 15 19:30:58.238690 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.238605 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxnzz\" (UniqueName: \"kubernetes.io/projected/3d33c40b-8842-42dd-b3c9-6fa76498d82e-kube-api-access-sxnzz\") pod \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " Jul 15 19:30:58.238690 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.238653 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-token\") pod \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " Jul 15 19:30:58.238690 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.238676 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d33c40b-8842-42dd-b3c9-6fa76498d82e-local-certs\") pod \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " Jul 15 19:30:58.238900 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.238700 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-kubeconfig\") pod \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " Jul 15 19:30:58.238900 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.238745 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-cacerts\") pod \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\" (UID: \"3d33c40b-8842-42dd-b3c9-6fa76498d82e\") " Jul 15 19:30:58.239066 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.239031 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "3d33c40b-8842-42dd-b3c9-6fa76498d82e" (UID: "3d33c40b-8842-42dd-b3c9-6fa76498d82e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:30:58.241045 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.241017 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d33c40b-8842-42dd-b3c9-6fa76498d82e-kube-api-access-sxnzz" (OuterVolumeSpecName: "kube-api-access-sxnzz") pod "3d33c40b-8842-42dd-b3c9-6fa76498d82e" (UID: "3d33c40b-8842-42dd-b3c9-6fa76498d82e"). InnerVolumeSpecName "kube-api-access-sxnzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:30:58.241118 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.241074 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-cacerts" (OuterVolumeSpecName: "cacerts") pod "3d33c40b-8842-42dd-b3c9-6fa76498d82e" (UID: "3d33c40b-8842-42dd-b3c9-6fa76498d82e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:30:58.241295 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.241269 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d33c40b-8842-42dd-b3c9-6fa76498d82e-local-certs" (OuterVolumeSpecName: "local-certs") pod "3d33c40b-8842-42dd-b3c9-6fa76498d82e" (UID: "3d33c40b-8842-42dd-b3c9-6fa76498d82e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:30:58.241380 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.241294 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d33c40b-8842-42dd-b3c9-6fa76498d82e" (UID: "3d33c40b-8842-42dd-b3c9-6fa76498d82e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:30:58.241380 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.241368 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "3d33c40b-8842-42dd-b3c9-6fa76498d82e" (UID: "3d33c40b-8842-42dd-b3c9-6fa76498d82e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:30:58.241498 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.241457 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "3d33c40b-8842-42dd-b3c9-6fa76498d82e" (UID: "3d33c40b-8842-42dd-b3c9-6fa76498d82e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:30:58.339535 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.339443 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-token\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:30:58.339535 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.339489 2564 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d33c40b-8842-42dd-b3c9-6fa76498d82e-local-certs\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:30:58.339535 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.339503 2564 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-kubeconfig\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:30:58.339535 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.339517 2564 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-cacerts\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:30:58.339535 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.339531 2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-csr-dns-cert\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:30:58.339821 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.339542 2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d33c40b-8842-42dd-b3c9-6fa76498d82e-istio-csr-ca-configmap\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:30:58.339821 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:58.339550 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxnzz\" (UniqueName: \"kubernetes.io/projected/3d33c40b-8842-42dd-b3c9-6fa76498d82e-kube-api-access-sxnzz\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:30:59.077860 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:59.077827 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" Jul 15 19:30:59.077860 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:59.077836 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8" event={"ID":"3d33c40b-8842-42dd-b3c9-6fa76498d82e","Type":"ContainerDied","Data":"b30c529b46a1847271cda1481149649995eedcdea263acff7b6cb746d171f436"} Jul 15 19:30:59.078309 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:59.077893 2564 scope.go:117] "RemoveContainer" containerID="b6a8ab1f3e2bde3e893b043132911c0b6a1c61380d0b15b3ac1969febb0b5aa5" Jul 15 19:30:59.079627 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:59.079606 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-76jrd" event={"ID":"c1430a43-7fd3-4989-ab2d-1047ea045c53","Type":"ContainerStarted","Data":"2e4d02587c3ae731067a333df3bd940e62a2cb4af774d81bebf15c75421b5fcc"} Jul 15 19:30:59.079842 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:59.079818 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nlmpm" Jul 15 19:30:59.079923 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:59.079864 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-76jrd" Jul 15 19:30:59.104221 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:59.104194 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8"] Jul 15 19:30:59.111821 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:59.111800 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-hjhq8"] Jul 15 19:30:59.140117 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:30:59.140067 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-76jrd" podStartSLOduration=2.985206817 podStartE2EDuration="6.140055332s" totalImagesPullingTime="3.154848515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:59.138277603 +0000 UTC m=+567.609752906" watchObservedRunningTime="2026-07-15 19:30:59.140055332 +0000 UTC m=+567.611530621" Jul 15 19:31:00.032633 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:00.032598 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-vxlh2" Jul 15 19:31:00.058771 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:00.058745 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d33c40b-8842-42dd-b3c9-6fa76498d82e" path="/var/lib/kubelet/pods/3d33c40b-8842-42dd-b3c9-6fa76498d82e/volumes" Jul 15 19:31:01.035004 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:01.034973 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-qtll6" Jul 15 19:31:09.926846 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:09.926811 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:31:20.006714 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:20.006683 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:31:30.014050 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:30.014012 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:31:30.087439 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:30.087399 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-76jrd" Jul 15 19:31:30.090366 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:30.090342 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nlmpm" Jul 15 19:31:32.091192 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:32.091083 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:31:42.919578 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:42.919525 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:31:43.015215 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.015176 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:31:43.078772 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.078731 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dzn4g"] Jul 15 19:31:43.079061 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.079044 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d33c40b-8842-42dd-b3c9-6fa76498d82e" containerName="discovery" Jul 15 19:31:43.079108 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.079066 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d33c40b-8842-42dd-b3c9-6fa76498d82e" containerName="discovery" Jul 15 19:31:43.079143 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.079130 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d33c40b-8842-42dd-b3c9-6fa76498d82e" containerName="discovery" Jul 15 19:31:43.081188 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.081163 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dzn4g" Jul 15 19:31:43.084525 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.084501 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-lr2l9\"" Jul 15 19:31:43.085036 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.085015 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 19:31:43.085341 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.085323 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 19:31:43.095674 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.095651 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dzn4g"] Jul 15 19:31:43.141927 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.141885 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:31:43.207055 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.206980 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9gdh\" (UniqueName: \"kubernetes.io/projected/0ca95536-5816-4166-8326-eef611f0b12b-kube-api-access-t9gdh\") pod \"kuadrant-operator-catalog-dzn4g\" (UID: \"0ca95536-5816-4166-8326-eef611f0b12b\") " pod="kuadrant-system/kuadrant-operator-catalog-dzn4g" Jul 15 19:31:43.308228 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.308198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9gdh\" (UniqueName: \"kubernetes.io/projected/0ca95536-5816-4166-8326-eef611f0b12b-kube-api-access-t9gdh\") pod \"kuadrant-operator-catalog-dzn4g\" (UID: \"0ca95536-5816-4166-8326-eef611f0b12b\") " pod="kuadrant-system/kuadrant-operator-catalog-dzn4g" Jul 15 19:31:43.323167 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.323144 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9gdh\" (UniqueName: \"kubernetes.io/projected/0ca95536-5816-4166-8326-eef611f0b12b-kube-api-access-t9gdh\") pod \"kuadrant-operator-catalog-dzn4g\" (UID: \"0ca95536-5816-4166-8326-eef611f0b12b\") " pod="kuadrant-system/kuadrant-operator-catalog-dzn4g" Jul 15 19:31:43.391354 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.391321 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dzn4g" Jul 15 19:31:43.425818 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.425777 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dzn4g"] Jul 15 19:31:43.524630 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.524595 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dzn4g"] Jul 15 19:31:43.529436 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:31:43.529406 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca95536_5816_4166_8326_eef611f0b12b.slice/crio-6a5c54023298c654a20c596711bc60135335dafeb6fac09b7037108416d7400a WatchSource:0}: Error finding container 6a5c54023298c654a20c596711bc60135335dafeb6fac09b7037108416d7400a: Status 404 returned error can't find the container with id 6a5c54023298c654a20c596711bc60135335dafeb6fac09b7037108416d7400a Jul 15 19:31:43.531179 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.531157 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:31:43.631025 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.630991 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t5wcq"] Jul 15 19:31:43.633949 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.633934 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-t5wcq" Jul 15 19:31:43.641969 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.641943 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t5wcq"] Jul 15 19:31:43.813798 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.813715 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drhg2\" (UniqueName: \"kubernetes.io/projected/1ab19b99-0b01-4170-beb7-a57491ee7378-kube-api-access-drhg2\") pod \"kuadrant-operator-catalog-t5wcq\" (UID: \"1ab19b99-0b01-4170-beb7-a57491ee7378\") " pod="kuadrant-system/kuadrant-operator-catalog-t5wcq" Jul 15 19:31:43.914138 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.914107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drhg2\" (UniqueName: \"kubernetes.io/projected/1ab19b99-0b01-4170-beb7-a57491ee7378-kube-api-access-drhg2\") pod \"kuadrant-operator-catalog-t5wcq\" (UID: \"1ab19b99-0b01-4170-beb7-a57491ee7378\") " pod="kuadrant-system/kuadrant-operator-catalog-t5wcq" Jul 15 19:31:43.922252 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.922227 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drhg2\" (UniqueName: \"kubernetes.io/projected/1ab19b99-0b01-4170-beb7-a57491ee7378-kube-api-access-drhg2\") pod \"kuadrant-operator-catalog-t5wcq\" (UID: \"1ab19b99-0b01-4170-beb7-a57491ee7378\") " pod="kuadrant-system/kuadrant-operator-catalog-t5wcq" Jul 15 19:31:43.943131 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:43.943097 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-t5wcq" Jul 15 19:31:44.075512 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:44.075439 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t5wcq"] Jul 15 19:31:44.077859 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:31:44.077831 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab19b99_0b01_4170_beb7_a57491ee7378.slice/crio-b0107036deb3e9460b868ee1492f5191ffc710a1619cdcb5a0d5b0bf09e77bd5 WatchSource:0}: Error finding container b0107036deb3e9460b868ee1492f5191ffc710a1619cdcb5a0d5b0bf09e77bd5: Status 404 returned error can't find the container with id b0107036deb3e9460b868ee1492f5191ffc710a1619cdcb5a0d5b0bf09e77bd5 Jul 15 19:31:44.260676 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:44.260645 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dzn4g" event={"ID":"0ca95536-5816-4166-8326-eef611f0b12b","Type":"ContainerStarted","Data":"6a5c54023298c654a20c596711bc60135335dafeb6fac09b7037108416d7400a"} Jul 15 19:31:44.261741 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:44.261716 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-t5wcq" event={"ID":"1ab19b99-0b01-4170-beb7-a57491ee7378","Type":"ContainerStarted","Data":"b0107036deb3e9460b868ee1492f5191ffc710a1619cdcb5a0d5b0bf09e77bd5"} Jul 15 19:31:47.276567 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:47.276526 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dzn4g" event={"ID":"0ca95536-5816-4166-8326-eef611f0b12b","Type":"ContainerStarted","Data":"dd4acd5458300a7a7e1ade8a361d5421183ec6111327c173e7f0833f12844978"} Jul 15 19:31:47.276995 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:47.276567 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-dzn4g" podUID="0ca95536-5816-4166-8326-eef611f0b12b" containerName="registry-server" containerID="cri-o://dd4acd5458300a7a7e1ade8a361d5421183ec6111327c173e7f0833f12844978" gracePeriod=2 Jul 15 19:31:47.278102 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:47.278081 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-t5wcq" event={"ID":"1ab19b99-0b01-4170-beb7-a57491ee7378","Type":"ContainerStarted","Data":"6dd8d6379f163e4b6696707005bf96206d3761899f69a62120ab08c1cab683b0"} Jul 15 19:31:47.293005 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:47.292968 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-dzn4g" podStartSLOduration=1.474846018 podStartE2EDuration="4.292954818s" totalImagesPullingTime="2.8181088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:31:47.291342834 +0000 UTC m=+615.762818124" watchObservedRunningTime="2026-07-15 19:31:47.292954818 +0000 UTC m=+615.764430107" Jul 15 19:31:47.307173 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:47.307136 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-t5wcq" podStartSLOduration=2.040988268 podStartE2EDuration="4.307123624s" totalImagesPullingTime="2.266135356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:31:47.30613106 +0000 UTC m=+615.777606351" watchObservedRunningTime="2026-07-15 19:31:47.307123624 +0000 UTC m=+615.778598912" Jul 15 19:31:47.516663 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:47.516635 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dzn4g" Jul 15 19:31:47.538958 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:47.538898 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9gdh\" (UniqueName: \"kubernetes.io/projected/0ca95536-5816-4166-8326-eef611f0b12b-kube-api-access-t9gdh\") pod \"0ca95536-5816-4166-8326-eef611f0b12b\" (UID: \"0ca95536-5816-4166-8326-eef611f0b12b\") " Jul 15 19:31:47.541114 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:47.541087 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca95536-5816-4166-8326-eef611f0b12b-kube-api-access-t9gdh" (OuterVolumeSpecName: "kube-api-access-t9gdh") pod "0ca95536-5816-4166-8326-eef611f0b12b" (UID: "0ca95536-5816-4166-8326-eef611f0b12b"). InnerVolumeSpecName "kube-api-access-t9gdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:31:47.639899 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:47.639864 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9gdh\" (UniqueName: \"kubernetes.io/projected/0ca95536-5816-4166-8326-eef611f0b12b-kube-api-access-t9gdh\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:31:48.283069 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:48.283031 2564 generic.go:356] "Generic (PLEG): container finished" podID="0ca95536-5816-4166-8326-eef611f0b12b" containerID="dd4acd5458300a7a7e1ade8a361d5421183ec6111327c173e7f0833f12844978" exitCode=0 Jul 15 19:31:48.283436 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:48.283076 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dzn4g" Jul 15 19:31:48.283436 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:48.283121 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dzn4g" event={"ID":"0ca95536-5816-4166-8326-eef611f0b12b","Type":"ContainerDied","Data":"dd4acd5458300a7a7e1ade8a361d5421183ec6111327c173e7f0833f12844978"} Jul 15 19:31:48.283436 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:48.283197 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dzn4g" event={"ID":"0ca95536-5816-4166-8326-eef611f0b12b","Type":"ContainerDied","Data":"6a5c54023298c654a20c596711bc60135335dafeb6fac09b7037108416d7400a"} Jul 15 19:31:48.283436 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:48.283222 2564 scope.go:117] "RemoveContainer" containerID="dd4acd5458300a7a7e1ade8a361d5421183ec6111327c173e7f0833f12844978" Jul 15 19:31:48.291941 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:48.291924 2564 scope.go:117] "RemoveContainer" containerID="dd4acd5458300a7a7e1ade8a361d5421183ec6111327c173e7f0833f12844978" Jul 15 19:31:48.292199 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:31:48.292183 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4acd5458300a7a7e1ade8a361d5421183ec6111327c173e7f0833f12844978\": container with ID starting with dd4acd5458300a7a7e1ade8a361d5421183ec6111327c173e7f0833f12844978 not found: ID does not exist" containerID="dd4acd5458300a7a7e1ade8a361d5421183ec6111327c173e7f0833f12844978" Jul 15 19:31:48.292263 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:48.292208 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4acd5458300a7a7e1ade8a361d5421183ec6111327c173e7f0833f12844978"} err="failed to get container status \"dd4acd5458300a7a7e1ade8a361d5421183ec6111327c173e7f0833f12844978\": rpc error: code = NotFound desc = could not find container \"dd4acd5458300a7a7e1ade8a361d5421183ec6111327c173e7f0833f12844978\": container with ID starting with dd4acd5458300a7a7e1ade8a361d5421183ec6111327c173e7f0833f12844978 not found: ID does not exist" Jul 15 19:31:48.298731 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:48.298695 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dzn4g"] Jul 15 19:31:48.300305 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:48.300285 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dzn4g"] Jul 15 19:31:50.059533 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:50.059496 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ca95536-5816-4166-8326-eef611f0b12b" path="/var/lib/kubelet/pods/0ca95536-5816-4166-8326-eef611f0b12b/volumes" Jul 15 19:31:53.944206 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:53.944118 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-t5wcq" Jul 15 19:31:53.944206 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:53.944175 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-t5wcq" Jul 15 19:31:53.966445 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:53.966420 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-t5wcq" Jul 15 19:31:54.331614 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:31:54.331588 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-t5wcq" Jul 15 19:32:03.638046 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:03.637967 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:03.717709 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:03.717674 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:05.387162 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:05.387132 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:05.443615 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:05.443584 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:11.346495 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:11.346453 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:14.010594 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:14.010560 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl"] Jul 15 19:32:14.010986 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:14.010869 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ca95536-5816-4166-8326-eef611f0b12b" containerName="registry-server" Jul 15 19:32:14.010986 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:14.010881 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ca95536-5816-4166-8326-eef611f0b12b" containerName="registry-server" Jul 15 19:32:14.010986 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:14.010941 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ca95536-5816-4166-8326-eef611f0b12b" containerName="registry-server" Jul 15 19:32:14.013858 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:14.013840 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl" Jul 15 19:32:14.019649 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:14.019622 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dqhhs\"" Jul 15 19:32:14.027522 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:14.027496 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl"] Jul 15 19:32:14.146848 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:14.146803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5pq8\" (UniqueName: \"kubernetes.io/projected/f43d00c9-c42a-4386-9803-08a3406fc8b6-kube-api-access-t5pq8\") pod \"limitador-operator-controller-manager-859d7b78bd-k8hvl\" (UID: \"f43d00c9-c42a-4386-9803-08a3406fc8b6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl" Jul 15 19:32:14.248340 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:14.248287 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5pq8\" (UniqueName: \"kubernetes.io/projected/f43d00c9-c42a-4386-9803-08a3406fc8b6-kube-api-access-t5pq8\") pod \"limitador-operator-controller-manager-859d7b78bd-k8hvl\" (UID: \"f43d00c9-c42a-4386-9803-08a3406fc8b6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl" Jul 15 19:32:14.260712 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:14.260645 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5pq8\" (UniqueName: \"kubernetes.io/projected/f43d00c9-c42a-4386-9803-08a3406fc8b6-kube-api-access-t5pq8\") pod \"limitador-operator-controller-manager-859d7b78bd-k8hvl\" (UID: \"f43d00c9-c42a-4386-9803-08a3406fc8b6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl" Jul 15 19:32:14.323265 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:14.323235 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl" Jul 15 19:32:14.462748 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:14.462714 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl"] Jul 15 19:32:14.464693 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:32:14.464664 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43d00c9_c42a_4386_9803_08a3406fc8b6.slice/crio-25b5eef2d43d638ba7046d1fab7ee86e2a3f264916d0e19aee301b281d29dd26 WatchSource:0}: Error finding container 25b5eef2d43d638ba7046d1fab7ee86e2a3f264916d0e19aee301b281d29dd26: Status 404 returned error can't find the container with id 25b5eef2d43d638ba7046d1fab7ee86e2a3f264916d0e19aee301b281d29dd26 Jul 15 19:32:14.735088 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:14.735052 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:15.049103 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:15.049033 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:15.149629 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:15.149596 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:15.256435 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:15.256401 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:15.397810 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:15.397745 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl" event={"ID":"f43d00c9-c42a-4386-9803-08a3406fc8b6","Type":"ContainerStarted","Data":"25b5eef2d43d638ba7046d1fab7ee86e2a3f264916d0e19aee301b281d29dd26"} Jul 15 19:32:15.563447 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:15.563406 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:17.408662 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:17.408627 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl" event={"ID":"f43d00c9-c42a-4386-9803-08a3406fc8b6","Type":"ContainerStarted","Data":"29e8aaff92bca70f836c0c55fa517c4ad82fd4be30be8b83e35faf5079442bad"} Jul 15 19:32:17.409057 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:17.408693 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl" Jul 15 19:32:17.425266 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:17.425222 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl" podStartSLOduration=2.424728052 podStartE2EDuration="4.42520647s" totalImagesPullingTime="2.000478418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:32:17.424055191 +0000 UTC m=+645.895530491" watchObservedRunningTime="2026-07-15 19:32:17.42520647 +0000 UTC m=+645.896681810" Jul 15 19:32:18.042431 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:18.042399 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-h52kx"] Jul 15 19:32:18.049429 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:18.049407 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h52kx" Jul 15 19:32:18.051828 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:18.051803 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-qzrkw\"" Jul 15 19:32:18.077497 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:18.065880 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-h52kx"] Jul 15 19:32:18.180520 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:18.180464 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbwj6\" (UniqueName: \"kubernetes.io/projected/c29b17f7-d92e-4aaa-a2a8-07aa4514f1a9-kube-api-access-tbwj6\") pod \"authorino-operator-75cc8d5cb7-h52kx\" (UID: \"c29b17f7-d92e-4aaa-a2a8-07aa4514f1a9\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-h52kx" Jul 15 19:32:18.281396 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:18.281361 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbwj6\" (UniqueName: \"kubernetes.io/projected/c29b17f7-d92e-4aaa-a2a8-07aa4514f1a9-kube-api-access-tbwj6\") pod \"authorino-operator-75cc8d5cb7-h52kx\" (UID: \"c29b17f7-d92e-4aaa-a2a8-07aa4514f1a9\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-h52kx" Jul 15 19:32:18.293676 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:18.293604 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbwj6\" (UniqueName: \"kubernetes.io/projected/c29b17f7-d92e-4aaa-a2a8-07aa4514f1a9-kube-api-access-tbwj6\") pod \"authorino-operator-75cc8d5cb7-h52kx\" (UID: \"c29b17f7-d92e-4aaa-a2a8-07aa4514f1a9\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-h52kx" Jul 15 19:32:18.364571 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:18.364533 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h52kx" Jul 15 19:32:18.529070 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:18.529040 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-h52kx"] Jul 15 19:32:18.529383 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:32:18.529284 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29b17f7_d92e_4aaa_a2a8_07aa4514f1a9.slice/crio-e422b93ba18a3804ba92f5d6337aab9f6cd3a7eab3a5868404ecbd629d74bb5a WatchSource:0}: Error finding container e422b93ba18a3804ba92f5d6337aab9f6cd3a7eab3a5868404ecbd629d74bb5a: Status 404 returned error can't find the container with id e422b93ba18a3804ba92f5d6337aab9f6cd3a7eab3a5868404ecbd629d74bb5a Jul 15 19:32:18.616392 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:18.616330 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:18.715579 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:18.715544 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:18.813314 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:18.813247 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:19.419782 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:19.419749 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h52kx" event={"ID":"c29b17f7-d92e-4aaa-a2a8-07aa4514f1a9","Type":"ContainerStarted","Data":"e422b93ba18a3804ba92f5d6337aab9f6cd3a7eab3a5868404ecbd629d74bb5a"} Jul 15 19:32:21.468326 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:21.468296 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:22.435952 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:22.435917 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h52kx" event={"ID":"c29b17f7-d92e-4aaa-a2a8-07aa4514f1a9","Type":"ContainerStarted","Data":"48aaba7e84138b412f65b4b73e1a099f61ad891931b0de6b43d313b833045b07"} Jul 15 19:32:22.436120 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:22.436041 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h52kx" Jul 15 19:32:22.455578 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:22.455515 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h52kx" podStartSLOduration=1.3106264570000001 podStartE2EDuration="4.455499262s" totalImagesPullingTime="3.144872805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:32:22.455499936 +0000 UTC m=+650.926975222" watchObservedRunningTime="2026-07-15 19:32:22.455499262 +0000 UTC m=+650.926974552" Jul 15 19:32:22.945071 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:22.945037 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9"] Jul 15 19:32:22.948306 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:22.948286 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:22.950551 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:22.950534 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m5c79\"" Jul 15 19:32:22.962678 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:22.962648 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9"] Jul 15 19:32:22.973163 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:22.973143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:23.023795 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:23.023760 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjzlg\" (UniqueName: \"kubernetes.io/projected/855aa335-649b-45b4-956a-270a471db7eb-kube-api-access-cjzlg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-75xh9\" (UID: \"855aa335-649b-45b4-956a-270a471db7eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:23.023942 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:23.023826 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/855aa335-649b-45b4-956a-270a471db7eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-75xh9\" (UID: \"855aa335-649b-45b4-956a-270a471db7eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:23.125136 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:23.125103 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/855aa335-649b-45b4-956a-270a471db7eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-75xh9\" (UID: \"855aa335-649b-45b4-956a-270a471db7eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:23.125244 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:23.125157 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjzlg\" (UniqueName: \"kubernetes.io/projected/855aa335-649b-45b4-956a-270a471db7eb-kube-api-access-cjzlg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-75xh9\" (UID: \"855aa335-649b-45b4-956a-270a471db7eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:23.125516 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:23.125496 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/855aa335-649b-45b4-956a-270a471db7eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-75xh9\" (UID: \"855aa335-649b-45b4-956a-270a471db7eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:23.134374 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:23.134352 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjzlg\" (UniqueName: \"kubernetes.io/projected/855aa335-649b-45b4-956a-270a471db7eb-kube-api-access-cjzlg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-75xh9\" (UID: \"855aa335-649b-45b4-956a-270a471db7eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:23.258627 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:23.258602 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:23.399561 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:23.399531 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9"] Jul 15 19:32:23.403446 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:32:23.403422 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855aa335_649b_45b4_956a_270a471db7eb.slice/crio-b082f522754e62f505d2b96238ad05f47f16ca1e8a6153a97b64117f18d1f083 WatchSource:0}: Error finding container b082f522754e62f505d2b96238ad05f47f16ca1e8a6153a97b64117f18d1f083: Status 404 returned error can't find the container with id b082f522754e62f505d2b96238ad05f47f16ca1e8a6153a97b64117f18d1f083 Jul 15 19:32:23.441609 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:23.441580 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" event={"ID":"855aa335-649b-45b4-956a-270a471db7eb","Type":"ContainerStarted","Data":"b082f522754e62f505d2b96238ad05f47f16ca1e8a6153a97b64117f18d1f083"} Jul 15 19:32:24.148089 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:24.148058 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:26.166854 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:26.166818 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:27.725621 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:27.725581 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:27.826676 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:27.826649 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:28.417334 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:28.417303 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl" Jul 15 19:32:30.477701 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:30.477664 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" event={"ID":"855aa335-649b-45b4-956a-270a471db7eb","Type":"ContainerStarted","Data":"e31b04327f72a9839e4643a40339160d810fff94c09af6bce5d8465be7c6ebc9"} Jul 15 19:32:30.478044 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:30.477814 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:30.497435 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:30.497388 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" podStartSLOduration=1.866353588 podStartE2EDuration="8.49737558s" totalImagesPullingTime="6.631021992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:32:30.494980536 +0000 UTC m=+658.966455825" watchObservedRunningTime="2026-07-15 19:32:30.49737558 +0000 UTC m=+658.968850868" Jul 15 19:32:31.726054 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:31.726022 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:31.780999 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:31.780970 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:32.129410 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:32.129322 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:32.189051 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:32.189016 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:32.974366 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:32.974334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:33.066776 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:33.066736 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:33.443791 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:33.443715 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h52kx" Jul 15 19:32:36.142339 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:36.142314 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:36.242865 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:36.242830 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:41.484110 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:41.484078 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:42.640994 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:42.640964 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg"] Jul 15 19:32:42.651539 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:42.651510 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" Jul 15 19:32:42.657376 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:42.657348 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg"] Jul 15 19:32:42.673454 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:42.673423 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:42.768232 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:42.768197 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:42.787809 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:42.787780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv92p\" (UniqueName: \"kubernetes.io/projected/b0c8422c-320f-4c06-8244-489d2a39a562-kube-api-access-zv92p\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4wsxg\" (UID: \"b0c8422c-320f-4c06-8244-489d2a39a562\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" Jul 15 19:32:42.787943 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:42.787815 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0c8422c-320f-4c06-8244-489d2a39a562-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4wsxg\" (UID: \"b0c8422c-320f-4c06-8244-489d2a39a562\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" Jul 15 19:32:42.888333 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:42.888291 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv92p\" (UniqueName: \"kubernetes.io/projected/b0c8422c-320f-4c06-8244-489d2a39a562-kube-api-access-zv92p\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4wsxg\" (UID: \"b0c8422c-320f-4c06-8244-489d2a39a562\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" Jul 15 19:32:42.888333 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:42.888336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0c8422c-320f-4c06-8244-489d2a39a562-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4wsxg\" (UID: \"b0c8422c-320f-4c06-8244-489d2a39a562\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" Jul 15 19:32:42.888765 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:42.888741 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0c8422c-320f-4c06-8244-489d2a39a562-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4wsxg\" (UID: \"b0c8422c-320f-4c06-8244-489d2a39a562\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" Jul 15 19:32:42.896705 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:42.896654 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv92p\" (UniqueName: \"kubernetes.io/projected/b0c8422c-320f-4c06-8244-489d2a39a562-kube-api-access-zv92p\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4wsxg\" (UID: \"b0c8422c-320f-4c06-8244-489d2a39a562\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" Jul 15 19:32:42.962674 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:42.962648 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" Jul 15 19:32:43.103644 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.103607 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg"] Jul 15 19:32:43.105829 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:32:43.105800 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c8422c_320f_4c06_8244_489d2a39a562.slice/crio-9ec8147ec7aeb989d582e62d9fcd33f003b9f7156345d2f1bffec36d2310bd1d WatchSource:0}: Error finding container 9ec8147ec7aeb989d582e62d9fcd33f003b9f7156345d2f1bffec36d2310bd1d: Status 404 returned error can't find the container with id 9ec8147ec7aeb989d582e62d9fcd33f003b9f7156345d2f1bffec36d2310bd1d Jul 15 19:32:43.389618 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.389586 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg"] Jul 15 19:32:43.396536 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.396510 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg"] Jul 15 19:32:43.408635 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.408573 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9"] Jul 15 19:32:43.408801 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.408781 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" podUID="855aa335-649b-45b4-956a-270a471db7eb" containerName="manager" containerID="cri-o://e31b04327f72a9839e4643a40339160d810fff94c09af6bce5d8465be7c6ebc9" gracePeriod=2 Jul 15 19:32:43.412928 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.412905 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv"] Jul 15 19:32:43.413392 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.413375 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c8422c-320f-4c06-8244-489d2a39a562" containerName="manager" Jul 15 19:32:43.413457 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.413395 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c8422c-320f-4c06-8244-489d2a39a562" containerName="manager" Jul 15 19:32:43.413552 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.413538 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0c8422c-320f-4c06-8244-489d2a39a562" containerName="manager" Jul 15 19:32:43.416608 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.416576 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" Jul 15 19:32:43.428211 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.428143 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9"] Jul 15 19:32:43.428586 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:43.428530 2564 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"855aa335-649b-45b4-956a-270a471db7eb\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-75xh9\": pods \"kuadrant-operator-controller-manager-8684c9ff49-75xh9\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:43.429957 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.429938 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv"] Jul 15 19:32:43.437655 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.437634 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5"] Jul 15 19:32:43.438190 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.438124 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="855aa335-649b-45b4-956a-270a471db7eb" containerName="manager" Jul 15 19:32:43.438190 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.438143 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="855aa335-649b-45b4-956a-270a471db7eb" containerName="manager" Jul 15 19:32:43.438349 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.438249 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="855aa335-649b-45b4-956a-270a471db7eb" containerName="manager" Jul 15 19:32:43.441446 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.441427 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl"] Jul 15 19:32:43.441576 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.441555 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" Jul 15 19:32:43.441668 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.441643 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl" podUID="f43d00c9-c42a-4386-9803-08a3406fc8b6" containerName="manager" containerID="cri-o://29e8aaff92bca70f836c0c55fa517c4ad82fd4be30be8b83e35faf5079442bad" gracePeriod=2 Jul 15 19:32:43.445019 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.444996 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl"] Jul 15 19:32:43.451968 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:43.451933 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-75xh9\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" podUID="855aa335-649b-45b4-956a-270a471db7eb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:43.455497 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.455453 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5"] Jul 15 19:32:43.471088 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:43.471063 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-k8hvl\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" podUID="f43d00c9-c42a-4386-9803-08a3406fc8b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl" Jul 15 19:32:43.472867 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:43.472836 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-75xh9\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" podUID="855aa335-649b-45b4-956a-270a471db7eb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:43.536438 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:43.536412 2564 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" container="manager" Jul 15 19:32:43.537845 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.537822 2564 generic.go:356] "Generic (PLEG): container finished" podID="855aa335-649b-45b4-956a-270a471db7eb" containerID="e31b04327f72a9839e4643a40339160d810fff94c09af6bce5d8465be7c6ebc9" exitCode=0 Jul 15 19:32:43.538228 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:43.538208 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-75xh9\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" podUID="855aa335-649b-45b4-956a-270a471db7eb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:43.539902 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:43.539882 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-4wsxg\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" podUID="b0c8422c-320f-4c06-8244-489d2a39a562" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" Jul 15 19:32:43.541586 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:43.541562 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-k8hvl\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" podUID="f43d00c9-c42a-4386-9803-08a3406fc8b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl" Jul 15 19:32:43.594541 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.594499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmgp5\" (UniqueName: \"kubernetes.io/projected/4cd06574-9589-405f-9dbf-7abf5fe3930a-kube-api-access-fmgp5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fbmrv\" (UID: \"4cd06574-9589-405f-9dbf-7abf5fe3930a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" Jul 15 19:32:43.594725 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.594611 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kcqr\" (UniqueName: \"kubernetes.io/projected/5bf83beb-0110-43a3-9b6a-0c3009a0d2be-kube-api-access-9kcqr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sgrd5\" (UID: \"5bf83beb-0110-43a3-9b6a-0c3009a0d2be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" Jul 15 19:32:43.594725 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.594641 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4cd06574-9589-405f-9dbf-7abf5fe3930a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fbmrv\" (UID: \"4cd06574-9589-405f-9dbf-7abf5fe3930a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" Jul 15 19:32:43.594725 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.594689 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bf83beb-0110-43a3-9b6a-0c3009a0d2be-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sgrd5\" (UID: \"5bf83beb-0110-43a3-9b6a-0c3009a0d2be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" Jul 15 19:32:43.668346 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.668321 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:43.670975 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:43.670945 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-k8hvl\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" podUID="f43d00c9-c42a-4386-9803-08a3406fc8b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl" Jul 15 19:32:43.672887 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:43.672862 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-75xh9\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" podUID="855aa335-649b-45b4-956a-270a471db7eb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:43.674645 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:43.674613 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-4wsxg\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" podUID="b0c8422c-320f-4c06-8244-489d2a39a562" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" Jul 15 19:32:43.676908 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.676890 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl" Jul 15 19:32:43.679002 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:43.678982 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-k8hvl\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" podUID="f43d00c9-c42a-4386-9803-08a3406fc8b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl" Jul 15 19:32:43.680960 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:43.680940 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-75xh9\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" podUID="855aa335-649b-45b4-956a-270a471db7eb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:43.682832 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:43.682814 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-4wsxg\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" podUID="b0c8422c-320f-4c06-8244-489d2a39a562" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" Jul 15 19:32:43.695932 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.695910 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4cd06574-9589-405f-9dbf-7abf5fe3930a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fbmrv\" (UID: \"4cd06574-9589-405f-9dbf-7abf5fe3930a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" Jul 15 19:32:43.696035 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.695945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bf83beb-0110-43a3-9b6a-0c3009a0d2be-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sgrd5\" (UID: \"5bf83beb-0110-43a3-9b6a-0c3009a0d2be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" Jul 15 19:32:43.696035 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.695990 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmgp5\" (UniqueName: \"kubernetes.io/projected/4cd06574-9589-405f-9dbf-7abf5fe3930a-kube-api-access-fmgp5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fbmrv\" (UID: \"4cd06574-9589-405f-9dbf-7abf5fe3930a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" Jul 15 19:32:43.696140 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.696101 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kcqr\" (UniqueName: \"kubernetes.io/projected/5bf83beb-0110-43a3-9b6a-0c3009a0d2be-kube-api-access-9kcqr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sgrd5\" (UID: \"5bf83beb-0110-43a3-9b6a-0c3009a0d2be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" Jul 15 19:32:43.696330 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.696297 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4cd06574-9589-405f-9dbf-7abf5fe3930a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fbmrv\" (UID: \"4cd06574-9589-405f-9dbf-7abf5fe3930a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" Jul 15 19:32:43.696379 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.696324 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bf83beb-0110-43a3-9b6a-0c3009a0d2be-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sgrd5\" (UID: \"5bf83beb-0110-43a3-9b6a-0c3009a0d2be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" Jul 15 19:32:43.714372 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.714343 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmgp5\" (UniqueName: \"kubernetes.io/projected/4cd06574-9589-405f-9dbf-7abf5fe3930a-kube-api-access-fmgp5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fbmrv\" (UID: \"4cd06574-9589-405f-9dbf-7abf5fe3930a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" Jul 15 19:32:43.719171 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.719143 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kcqr\" (UniqueName: \"kubernetes.io/projected/5bf83beb-0110-43a3-9b6a-0c3009a0d2be-kube-api-access-9kcqr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sgrd5\" (UID: \"5bf83beb-0110-43a3-9b6a-0c3009a0d2be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" Jul 15 19:32:43.730019 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.729997 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" Jul 15 19:32:43.796717 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.796687 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5pq8\" (UniqueName: \"kubernetes.io/projected/f43d00c9-c42a-4386-9803-08a3406fc8b6-kube-api-access-t5pq8\") pod \"f43d00c9-c42a-4386-9803-08a3406fc8b6\" (UID: \"f43d00c9-c42a-4386-9803-08a3406fc8b6\") " Jul 15 19:32:43.796865 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.796732 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjzlg\" (UniqueName: \"kubernetes.io/projected/855aa335-649b-45b4-956a-270a471db7eb-kube-api-access-cjzlg\") pod \"855aa335-649b-45b4-956a-270a471db7eb\" (UID: \"855aa335-649b-45b4-956a-270a471db7eb\") " Jul 15 19:32:43.796865 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.796798 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/855aa335-649b-45b4-956a-270a471db7eb-extensions-socket-volume\") pod \"855aa335-649b-45b4-956a-270a471db7eb\" (UID: \"855aa335-649b-45b4-956a-270a471db7eb\") " Jul 15 19:32:43.797432 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.797241 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855aa335-649b-45b4-956a-270a471db7eb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "855aa335-649b-45b4-956a-270a471db7eb" (UID: "855aa335-649b-45b4-956a-270a471db7eb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:32:43.798946 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.798916 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43d00c9-c42a-4386-9803-08a3406fc8b6-kube-api-access-t5pq8" (OuterVolumeSpecName: "kube-api-access-t5pq8") pod "f43d00c9-c42a-4386-9803-08a3406fc8b6" (UID: "f43d00c9-c42a-4386-9803-08a3406fc8b6"). InnerVolumeSpecName "kube-api-access-t5pq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:32:43.798946 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.798932 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/855aa335-649b-45b4-956a-270a471db7eb-kube-api-access-cjzlg" (OuterVolumeSpecName: "kube-api-access-cjzlg") pod "855aa335-649b-45b4-956a-270a471db7eb" (UID: "855aa335-649b-45b4-956a-270a471db7eb"). InnerVolumeSpecName "kube-api-access-cjzlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:32:43.848880 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.848847 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" Jul 15 19:32:43.874708 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.874676 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv"] Jul 15 19:32:43.878511 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:32:43.878458 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd06574_9589_405f_9dbf_7abf5fe3930a.slice/crio-633fe7e39a8a4a3b70142aa76a940a3ffd58126c8b44f2157333b383e6d4fecc WatchSource:0}: Error finding container 633fe7e39a8a4a3b70142aa76a940a3ffd58126c8b44f2157333b383e6d4fecc: Status 404 returned error can't find the container with id 633fe7e39a8a4a3b70142aa76a940a3ffd58126c8b44f2157333b383e6d4fecc Jul 15 19:32:43.897996 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.897975 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5pq8\" (UniqueName: \"kubernetes.io/projected/f43d00c9-c42a-4386-9803-08a3406fc8b6-kube-api-access-t5pq8\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:32:43.898101 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.898001 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjzlg\" (UniqueName: \"kubernetes.io/projected/855aa335-649b-45b4-956a-270a471db7eb-kube-api-access-cjzlg\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:32:43.898101 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.898016 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/855aa335-649b-45b4-956a-270a471db7eb-extensions-socket-volume\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:32:43.999179 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:43.999153 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5"] Jul 15 19:32:44.001615 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:32:44.001589 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf83beb_0110_43a3_9b6a_0c3009a0d2be.slice/crio-03d66e3ddf5300db3c58c9bfd823372d47a443399ce7b3aa10c422a170985006 WatchSource:0}: Error finding container 03d66e3ddf5300db3c58c9bfd823372d47a443399ce7b3aa10c422a170985006: Status 404 returned error can't find the container with id 03d66e3ddf5300db3c58c9bfd823372d47a443399ce7b3aa10c422a170985006 Jul 15 19:32:44.059684 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.059657 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="855aa335-649b-45b4-956a-270a471db7eb" path="/var/lib/kubelet/pods/855aa335-649b-45b4-956a-270a471db7eb/volumes" Jul 15 19:32:44.060050 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.060034 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f43d00c9-c42a-4386-9803-08a3406fc8b6" path="/var/lib/kubelet/pods/f43d00c9-c42a-4386-9803-08a3406fc8b6/volumes" Jul 15 19:32:44.544843 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.544803 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" event={"ID":"5bf83beb-0110-43a3-9b6a-0c3009a0d2be","Type":"ContainerStarted","Data":"189349b8dfe758b0bbcd8d329a5523ac9360606f4c50be716fb6c6c5d280cc5c"} Jul 15 19:32:44.545033 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.544862 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" event={"ID":"5bf83beb-0110-43a3-9b6a-0c3009a0d2be","Type":"ContainerStarted","Data":"03d66e3ddf5300db3c58c9bfd823372d47a443399ce7b3aa10c422a170985006"} Jul 15 19:32:44.545033 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.544889 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" Jul 15 19:32:44.546114 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.546095 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-75xh9" Jul 15 19:32:44.546210 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.546097 2564 scope.go:117] "RemoveContainer" containerID="e31b04327f72a9839e4643a40339160d810fff94c09af6bce5d8465be7c6ebc9" Jul 15 19:32:44.547998 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.547966 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" event={"ID":"4cd06574-9589-405f-9dbf-7abf5fe3930a","Type":"ContainerStarted","Data":"64213a7d48d10de8ca5f68184e12b1cb3701cc8d93f15154df99067ad138468e"} Jul 15 19:32:44.548110 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.548030 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" event={"ID":"4cd06574-9589-405f-9dbf-7abf5fe3930a","Type":"ContainerStarted","Data":"633fe7e39a8a4a3b70142aa76a940a3ffd58126c8b44f2157333b383e6d4fecc"} Jul 15 19:32:44.548110 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.548061 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" Jul 15 19:32:44.550059 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.550026 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" podUID="b0c8422c-320f-4c06-8244-489d2a39a562" containerName="manager" containerID="cri-o://57c96102d5f63ddf302c2abd00190c64b13b4194326a978a301b00653e3898f0" gracePeriod=2 Jul 15 19:32:44.551491 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.551454 2564 generic.go:356] "Generic (PLEG): container finished" podID="f43d00c9-c42a-4386-9803-08a3406fc8b6" containerID="29e8aaff92bca70f836c0c55fa517c4ad82fd4be30be8b83e35faf5079442bad" exitCode=0 Jul 15 19:32:44.551572 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.551525 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k8hvl" Jul 15 19:32:44.556729 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.556714 2564 scope.go:117] "RemoveContainer" containerID="29e8aaff92bca70f836c0c55fa517c4ad82fd4be30be8b83e35faf5079442bad" Jul 15 19:32:44.568374 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.568330 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" podStartSLOduration=1.5683177270000002 podStartE2EDuration="1.568317727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:32:44.566871627 +0000 UTC m=+673.038346918" watchObservedRunningTime="2026-07-15 19:32:44.568317727 +0000 UTC m=+673.039792994" Jul 15 19:32:44.586036 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.585990 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" podStartSLOduration=1.5859775470000002 podStartE2EDuration="1.585977547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:32:44.584056143 +0000 UTC m=+673.055531434" watchObservedRunningTime="2026-07-15 19:32:44.585977547 +0000 UTC m=+673.057452883" Jul 15 19:32:44.621263 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.621246 2564 scope.go:117] "RemoveContainer" containerID="29e8aaff92bca70f836c0c55fa517c4ad82fd4be30be8b83e35faf5079442bad" Jul 15 19:32:44.621584 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:44.621564 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e8aaff92bca70f836c0c55fa517c4ad82fd4be30be8b83e35faf5079442bad\": container with ID starting with 29e8aaff92bca70f836c0c55fa517c4ad82fd4be30be8b83e35faf5079442bad not found: ID does not exist" containerID="29e8aaff92bca70f836c0c55fa517c4ad82fd4be30be8b83e35faf5079442bad" Jul 15 19:32:44.621624 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.621594 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e8aaff92bca70f836c0c55fa517c4ad82fd4be30be8b83e35faf5079442bad"} err="failed to get container status \"29e8aaff92bca70f836c0c55fa517c4ad82fd4be30be8b83e35faf5079442bad\": rpc error: code = NotFound desc = could not find container \"29e8aaff92bca70f836c0c55fa517c4ad82fd4be30be8b83e35faf5079442bad\": container with ID starting with 29e8aaff92bca70f836c0c55fa517c4ad82fd4be30be8b83e35faf5079442bad not found: ID does not exist" Jul 15 19:32:44.786966 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.786945 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" Jul 15 19:32:44.789118 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:44.789095 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-4wsxg\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" podUID="b0c8422c-320f-4c06-8244-489d2a39a562" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" Jul 15 19:32:44.805407 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.805364 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0c8422c-320f-4c06-8244-489d2a39a562-extensions-socket-volume\") pod \"b0c8422c-320f-4c06-8244-489d2a39a562\" (UID: \"b0c8422c-320f-4c06-8244-489d2a39a562\") " Jul 15 19:32:44.805730 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.805709 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c8422c-320f-4c06-8244-489d2a39a562-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b0c8422c-320f-4c06-8244-489d2a39a562" (UID: "b0c8422c-320f-4c06-8244-489d2a39a562"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:32:44.905894 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.905860 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv92p\" (UniqueName: \"kubernetes.io/projected/b0c8422c-320f-4c06-8244-489d2a39a562-kube-api-access-zv92p\") pod \"b0c8422c-320f-4c06-8244-489d2a39a562\" (UID: \"b0c8422c-320f-4c06-8244-489d2a39a562\") " Jul 15 19:32:44.906067 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.906052 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0c8422c-320f-4c06-8244-489d2a39a562-extensions-socket-volume\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:32:44.907983 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:44.907951 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c8422c-320f-4c06-8244-489d2a39a562-kube-api-access-zv92p" (OuterVolumeSpecName: "kube-api-access-zv92p") pod "b0c8422c-320f-4c06-8244-489d2a39a562" (UID: "b0c8422c-320f-4c06-8244-489d2a39a562"). InnerVolumeSpecName "kube-api-access-zv92p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:32:45.006535 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:45.006505 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv92p\" (UniqueName: \"kubernetes.io/projected/b0c8422c-320f-4c06-8244-489d2a39a562-kube-api-access-zv92p\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:32:45.559834 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:45.559804 2564 generic.go:356] "Generic (PLEG): container finished" podID="b0c8422c-320f-4c06-8244-489d2a39a562" containerID="57c96102d5f63ddf302c2abd00190c64b13b4194326a978a301b00653e3898f0" exitCode=0 Jul 15 19:32:45.560008 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:45.559847 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" Jul 15 19:32:45.560008 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:45.559887 2564 scope.go:117] "RemoveContainer" containerID="57c96102d5f63ddf302c2abd00190c64b13b4194326a978a301b00653e3898f0" Jul 15 19:32:45.562214 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:45.562171 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-4wsxg\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" podUID="b0c8422c-320f-4c06-8244-489d2a39a562" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" Jul 15 19:32:45.569259 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:45.569241 2564 scope.go:117] "RemoveContainer" containerID="57c96102d5f63ddf302c2abd00190c64b13b4194326a978a301b00653e3898f0" Jul 15 19:32:45.569527 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:45.569498 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c96102d5f63ddf302c2abd00190c64b13b4194326a978a301b00653e3898f0\": container with ID starting with 57c96102d5f63ddf302c2abd00190c64b13b4194326a978a301b00653e3898f0 not found: ID does not exist" containerID="57c96102d5f63ddf302c2abd00190c64b13b4194326a978a301b00653e3898f0" Jul 15 19:32:45.569615 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:45.569535 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c96102d5f63ddf302c2abd00190c64b13b4194326a978a301b00653e3898f0"} err="failed to get container status \"57c96102d5f63ddf302c2abd00190c64b13b4194326a978a301b00653e3898f0\": rpc error: code = NotFound desc = could not find container \"57c96102d5f63ddf302c2abd00190c64b13b4194326a978a301b00653e3898f0\": container with ID starting with 57c96102d5f63ddf302c2abd00190c64b13b4194326a978a301b00653e3898f0 not found: ID does not exist" Jul 15 19:32:45.570414 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:45.570391 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-4wsxg\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" podUID="b0c8422c-320f-4c06-8244-489d2a39a562" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4wsxg" Jul 15 19:32:46.065042 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:46.065007 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0c8422c-320f-4c06-8244-489d2a39a562" path="/var/lib/kubelet/pods/b0c8422c-320f-4c06-8244-489d2a39a562/volumes" Jul 15 19:32:46.976500 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:46.976445 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:47.087861 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:47.087829 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:52.373001 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:52.372958 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:52.473735 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:52.473690 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:52.969899 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:52.969867 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:53.024906 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:53.024875 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:55.565039 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:55.565012 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" Jul 15 19:32:55.565545 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:55.565094 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" Jul 15 19:32:55.776740 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:55.776710 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5"] Jul 15 19:32:55.776920 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:55.776891 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" podUID="5bf83beb-0110-43a3-9b6a-0c3009a0d2be" containerName="manager" containerID="cri-o://189349b8dfe758b0bbcd8d329a5523ac9360606f4c50be716fb6c6c5d280cc5c" gracePeriod=10 Jul 15 19:32:56.029666 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.029646 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" Jul 15 19:32:56.073747 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.073715 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps"] Jul 15 19:32:56.074031 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.074018 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f43d00c9-c42a-4386-9803-08a3406fc8b6" containerName="manager" Jul 15 19:32:56.074080 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.074036 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f43d00c9-c42a-4386-9803-08a3406fc8b6" containerName="manager" Jul 15 19:32:56.074080 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.074057 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf83beb-0110-43a3-9b6a-0c3009a0d2be" containerName="manager" Jul 15 19:32:56.074080 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.074063 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf83beb-0110-43a3-9b6a-0c3009a0d2be" containerName="manager" Jul 15 19:32:56.074193 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.074119 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f43d00c9-c42a-4386-9803-08a3406fc8b6" containerName="manager" Jul 15 19:32:56.074193 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.074131 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf83beb-0110-43a3-9b6a-0c3009a0d2be" containerName="manager" Jul 15 19:32:56.077353 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.077327 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" Jul 15 19:32:56.088817 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.088795 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps"] Jul 15 19:32:56.090398 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.090375 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bf83beb-0110-43a3-9b6a-0c3009a0d2be-extensions-socket-volume\") pod \"5bf83beb-0110-43a3-9b6a-0c3009a0d2be\" (UID: \"5bf83beb-0110-43a3-9b6a-0c3009a0d2be\") " Jul 15 19:32:56.090516 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.090443 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kcqr\" (UniqueName: \"kubernetes.io/projected/5bf83beb-0110-43a3-9b6a-0c3009a0d2be-kube-api-access-9kcqr\") pod \"5bf83beb-0110-43a3-9b6a-0c3009a0d2be\" (UID: \"5bf83beb-0110-43a3-9b6a-0c3009a0d2be\") " Jul 15 19:32:56.090598 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.090582 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8c07e9e-5cfd-4ab6-b28b-a015c53363c8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-sx5ps\" (UID: \"f8c07e9e-5cfd-4ab6-b28b-a015c53363c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" Jul 15 19:32:56.090658 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.090633 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpglq\" (UniqueName: \"kubernetes.io/projected/f8c07e9e-5cfd-4ab6-b28b-a015c53363c8-kube-api-access-gpglq\") pod \"kuadrant-operator-controller-manager-86878c769f-sx5ps\" (UID: \"f8c07e9e-5cfd-4ab6-b28b-a015c53363c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" Jul 15 19:32:56.090761 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.090726 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf83beb-0110-43a3-9b6a-0c3009a0d2be-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5bf83beb-0110-43a3-9b6a-0c3009a0d2be" (UID: "5bf83beb-0110-43a3-9b6a-0c3009a0d2be"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:32:56.092645 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.092611 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf83beb-0110-43a3-9b6a-0c3009a0d2be-kube-api-access-9kcqr" (OuterVolumeSpecName: "kube-api-access-9kcqr") pod "5bf83beb-0110-43a3-9b6a-0c3009a0d2be" (UID: "5bf83beb-0110-43a3-9b6a-0c3009a0d2be"). InnerVolumeSpecName "kube-api-access-9kcqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:32:56.192059 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.191979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8c07e9e-5cfd-4ab6-b28b-a015c53363c8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-sx5ps\" (UID: \"f8c07e9e-5cfd-4ab6-b28b-a015c53363c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" Jul 15 19:32:56.192059 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.192042 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpglq\" (UniqueName: \"kubernetes.io/projected/f8c07e9e-5cfd-4ab6-b28b-a015c53363c8-kube-api-access-gpglq\") pod \"kuadrant-operator-controller-manager-86878c769f-sx5ps\" (UID: \"f8c07e9e-5cfd-4ab6-b28b-a015c53363c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" Jul 15 19:32:56.192238 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.192079 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kcqr\" (UniqueName: \"kubernetes.io/projected/5bf83beb-0110-43a3-9b6a-0c3009a0d2be-kube-api-access-9kcqr\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:32:56.192238 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.192095 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bf83beb-0110-43a3-9b6a-0c3009a0d2be-extensions-socket-volume\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:32:56.192430 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.192411 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8c07e9e-5cfd-4ab6-b28b-a015c53363c8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-sx5ps\" (UID: \"f8c07e9e-5cfd-4ab6-b28b-a015c53363c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" Jul 15 19:32:56.204496 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.204439 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpglq\" (UniqueName: \"kubernetes.io/projected/f8c07e9e-5cfd-4ab6-b28b-a015c53363c8-kube-api-access-gpglq\") pod \"kuadrant-operator-controller-manager-86878c769f-sx5ps\" (UID: \"f8c07e9e-5cfd-4ab6-b28b-a015c53363c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" Jul 15 19:32:56.387529 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.387490 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" Jul 15 19:32:56.522178 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.522147 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps"] Jul 15 19:32:56.526004 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:32:56.525978 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c07e9e_5cfd_4ab6_b28b_a015c53363c8.slice/crio-abbf71801bf314475d8e5aa1552dcc29aaa56f4c20a8de6dec686ae8547533f6 WatchSource:0}: Error finding container abbf71801bf314475d8e5aa1552dcc29aaa56f4c20a8de6dec686ae8547533f6: Status 404 returned error can't find the container with id abbf71801bf314475d8e5aa1552dcc29aaa56f4c20a8de6dec686ae8547533f6 Jul 15 19:32:56.613251 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.613218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" event={"ID":"f8c07e9e-5cfd-4ab6-b28b-a015c53363c8","Type":"ContainerStarted","Data":"7dfc4b6e60d4e110ac669d0b74bb0cec3bb3b94ab0a39ba4bddef07646b081cc"} Jul 15 19:32:56.613706 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.613279 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" Jul 15 19:32:56.613706 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.613296 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" event={"ID":"f8c07e9e-5cfd-4ab6-b28b-a015c53363c8","Type":"ContainerStarted","Data":"abbf71801bf314475d8e5aa1552dcc29aaa56f4c20a8de6dec686ae8547533f6"} Jul 15 19:32:56.614584 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.614563 2564 generic.go:356] "Generic (PLEG): container finished" podID="5bf83beb-0110-43a3-9b6a-0c3009a0d2be" containerID="189349b8dfe758b0bbcd8d329a5523ac9360606f4c50be716fb6c6c5d280cc5c" exitCode=0 Jul 15 19:32:56.614678 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.614609 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" Jul 15 19:32:56.614718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.614613 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" event={"ID":"5bf83beb-0110-43a3-9b6a-0c3009a0d2be","Type":"ContainerDied","Data":"189349b8dfe758b0bbcd8d329a5523ac9360606f4c50be716fb6c6c5d280cc5c"} Jul 15 19:32:56.614718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.614710 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5" event={"ID":"5bf83beb-0110-43a3-9b6a-0c3009a0d2be","Type":"ContainerDied","Data":"03d66e3ddf5300db3c58c9bfd823372d47a443399ce7b3aa10c422a170985006"} Jul 15 19:32:56.614801 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.614730 2564 scope.go:117] "RemoveContainer" containerID="189349b8dfe758b0bbcd8d329a5523ac9360606f4c50be716fb6c6c5d280cc5c" Jul 15 19:32:56.624208 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.624163 2564 scope.go:117] "RemoveContainer" containerID="189349b8dfe758b0bbcd8d329a5523ac9360606f4c50be716fb6c6c5d280cc5c" Jul 15 19:32:56.629005 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:32:56.626107 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"189349b8dfe758b0bbcd8d329a5523ac9360606f4c50be716fb6c6c5d280cc5c\": container with ID starting with 189349b8dfe758b0bbcd8d329a5523ac9360606f4c50be716fb6c6c5d280cc5c not found: ID does not exist" containerID="189349b8dfe758b0bbcd8d329a5523ac9360606f4c50be716fb6c6c5d280cc5c" Jul 15 19:32:56.629005 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.626150 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"189349b8dfe758b0bbcd8d329a5523ac9360606f4c50be716fb6c6c5d280cc5c"} err="failed to get container status \"189349b8dfe758b0bbcd8d329a5523ac9360606f4c50be716fb6c6c5d280cc5c\": rpc error: code = NotFound desc = could not find container \"189349b8dfe758b0bbcd8d329a5523ac9360606f4c50be716fb6c6c5d280cc5c\": container with ID starting with 189349b8dfe758b0bbcd8d329a5523ac9360606f4c50be716fb6c6c5d280cc5c not found: ID does not exist" Jul 15 19:32:56.636995 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.636954 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" podStartSLOduration=0.636941366 podStartE2EDuration="636.941366ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:32:56.635901221 +0000 UTC m=+685.107376510" watchObservedRunningTime="2026-07-15 19:32:56.636941366 +0000 UTC m=+685.108416654" Jul 15 19:32:56.651443 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.651421 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5"] Jul 15 19:32:56.663357 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.663336 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgrd5"] Jul 15 19:32:56.752748 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.752672 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:56.859273 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:56.859237 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:58.059178 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:32:58.059145 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bf83beb-0110-43a3-9b6a-0c3009a0d2be" path="/var/lib/kubelet/pods/5bf83beb-0110-43a3-9b6a-0c3009a0d2be/volumes" Jul 15 19:33:02.744421 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:02.744385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:02.798909 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:02.798879 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:04.557401 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:04.557369 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:04.617364 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:04.617325 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:07.623498 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:07.623450 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" Jul 15 19:33:07.675052 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:07.675023 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv"] Jul 15 19:33:07.675225 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:07.675204 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" podUID="4cd06574-9589-405f-9dbf-7abf5fe3930a" containerName="manager" containerID="cri-o://64213a7d48d10de8ca5f68184e12b1cb3701cc8d93f15154df99067ad138468e" gracePeriod=10 Jul 15 19:33:07.922784 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:07.922756 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" Jul 15 19:33:07.981131 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:07.981095 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4cd06574-9589-405f-9dbf-7abf5fe3930a-extensions-socket-volume\") pod \"4cd06574-9589-405f-9dbf-7abf5fe3930a\" (UID: \"4cd06574-9589-405f-9dbf-7abf5fe3930a\") " Jul 15 19:33:07.981287 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:07.981216 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmgp5\" (UniqueName: \"kubernetes.io/projected/4cd06574-9589-405f-9dbf-7abf5fe3930a-kube-api-access-fmgp5\") pod \"4cd06574-9589-405f-9dbf-7abf5fe3930a\" (UID: \"4cd06574-9589-405f-9dbf-7abf5fe3930a\") " Jul 15 19:33:07.981535 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:07.981508 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd06574-9589-405f-9dbf-7abf5fe3930a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4cd06574-9589-405f-9dbf-7abf5fe3930a" (UID: "4cd06574-9589-405f-9dbf-7abf5fe3930a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:33:07.983331 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:07.983305 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd06574-9589-405f-9dbf-7abf5fe3930a-kube-api-access-fmgp5" (OuterVolumeSpecName: "kube-api-access-fmgp5") pod "4cd06574-9589-405f-9dbf-7abf5fe3930a" (UID: "4cd06574-9589-405f-9dbf-7abf5fe3930a"). InnerVolumeSpecName "kube-api-access-fmgp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:33:08.082035 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:08.082007 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4cd06574-9589-405f-9dbf-7abf5fe3930a-extensions-socket-volume\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:33:08.082035 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:08.082031 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmgp5\" (UniqueName: \"kubernetes.io/projected/4cd06574-9589-405f-9dbf-7abf5fe3930a-kube-api-access-fmgp5\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:33:08.670749 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:08.670706 2564 generic.go:356] "Generic (PLEG): container finished" podID="4cd06574-9589-405f-9dbf-7abf5fe3930a" containerID="64213a7d48d10de8ca5f68184e12b1cb3701cc8d93f15154df99067ad138468e" exitCode=0 Jul 15 19:33:08.671208 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:08.670759 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" Jul 15 19:33:08.671208 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:08.670774 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" event={"ID":"4cd06574-9589-405f-9dbf-7abf5fe3930a","Type":"ContainerDied","Data":"64213a7d48d10de8ca5f68184e12b1cb3701cc8d93f15154df99067ad138468e"} Jul 15 19:33:08.671208 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:08.670829 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv" event={"ID":"4cd06574-9589-405f-9dbf-7abf5fe3930a","Type":"ContainerDied","Data":"633fe7e39a8a4a3b70142aa76a940a3ffd58126c8b44f2157333b383e6d4fecc"} Jul 15 19:33:08.671208 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:08.670857 2564 scope.go:117] "RemoveContainer" containerID="64213a7d48d10de8ca5f68184e12b1cb3701cc8d93f15154df99067ad138468e" Jul 15 19:33:08.679611 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:08.679438 2564 scope.go:117] "RemoveContainer" containerID="64213a7d48d10de8ca5f68184e12b1cb3701cc8d93f15154df99067ad138468e" Jul 15 19:33:08.679713 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:33:08.679696 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64213a7d48d10de8ca5f68184e12b1cb3701cc8d93f15154df99067ad138468e\": container with ID starting with 64213a7d48d10de8ca5f68184e12b1cb3701cc8d93f15154df99067ad138468e not found: ID does not exist" containerID="64213a7d48d10de8ca5f68184e12b1cb3701cc8d93f15154df99067ad138468e" Jul 15 19:33:08.679756 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:08.679723 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64213a7d48d10de8ca5f68184e12b1cb3701cc8d93f15154df99067ad138468e"} err="failed to get container status \"64213a7d48d10de8ca5f68184e12b1cb3701cc8d93f15154df99067ad138468e\": rpc error: code = NotFound desc = could not find container \"64213a7d48d10de8ca5f68184e12b1cb3701cc8d93f15154df99067ad138468e\": container with ID starting with 64213a7d48d10de8ca5f68184e12b1cb3701cc8d93f15154df99067ad138468e not found: ID does not exist" Jul 15 19:33:08.696240 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:08.696217 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv"] Jul 15 19:33:08.702627 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:08.702605 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbmrv"] Jul 15 19:33:10.059380 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:10.059345 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cd06574-9589-405f-9dbf-7abf5fe3930a" path="/var/lib/kubelet/pods/4cd06574-9589-405f-9dbf-7abf5fe3930a/volumes" Jul 15 19:33:10.886310 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:10.886270 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:12.267002 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.266968 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd"] Jul 15 19:33:12.267374 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.267297 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cd06574-9589-405f-9dbf-7abf5fe3930a" containerName="manager" Jul 15 19:33:12.267374 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.267309 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cd06574-9589-405f-9dbf-7abf5fe3930a" containerName="manager" Jul 15 19:33:12.267447 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.267383 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cd06574-9589-405f-9dbf-7abf5fe3930a" containerName="manager" Jul 15 19:33:12.270504 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.270464 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.273161 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.273132 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sv688\"" Jul 15 19:33:12.295302 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.295279 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd"] Jul 15 19:33:12.305886 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.305858 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:12.315528 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.315495 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.315626 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.315529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.315626 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.315549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.315705 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.315624 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.315705 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.315658 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.315705 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.315679 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz5lm\" (UniqueName: \"kubernetes.io/projected/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-kube-api-access-cz5lm\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.315809 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.315751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.315809 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.315787 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.315877 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.315830 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.417129 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.417087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.417129 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.417126 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.417377 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.417155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.417377 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.417182 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.417377 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.417200 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.417377 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.417219 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.417377 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.417249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.417377 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.417271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.417377 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.417299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz5lm\" (UniqueName: \"kubernetes.io/projected/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-kube-api-access-cz5lm\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.417765 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.417622 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.417765 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.417726 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.417868 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.417789 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.417868 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.417849 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.418135 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.418108 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.419619 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.419596 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.419928 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.419908 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.427513 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.427488 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz5lm\" (UniqueName: \"kubernetes.io/projected/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-kube-api-access-cz5lm\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.427513 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.427506 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-srxrd\" (UID: \"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.580176 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.580083 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:12.726555 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.721683 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd"] Jul 15 19:33:12.727936 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.727907 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 19:33:12.728055 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.727980 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 19:33:12.728055 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.728019 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 19:33:12.922128 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:12.922024 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:13.697388 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:13.697355 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" event={"ID":"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd","Type":"ContainerStarted","Data":"a4c5bc43424edb107decb594cedae3f5014bad965ce76b6b694f90ead3e4efa5"} Jul 15 19:33:13.697885 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:13.697410 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" event={"ID":"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd","Type":"ContainerStarted","Data":"605c776eecdee7c3e4db935af057638b05637890bf9fcc46ba36f14ddf00d39e"} Jul 15 19:33:13.717687 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:13.717639 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" podStartSLOduration=1.717626737 podStartE2EDuration="1.717626737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:33:13.716394664 +0000 UTC m=+702.187869966" watchObservedRunningTime="2026-07-15 19:33:13.717626737 +0000 UTC m=+702.189102026" Jul 15 19:33:14.580964 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:14.580933 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:14.585736 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:14.585713 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:14.702212 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:14.702180 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:14.703162 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:14.703145 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:33:24.578174 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:24.578094 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:24.583810 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:24.583787 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-67nnj"] Jul 15 19:33:24.586169 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:24.586148 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" Jul 15 19:33:24.588130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:24.588102 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 19:33:24.588218 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:24.588171 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-drc2n\"" Jul 15 19:33:24.594837 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:24.594813 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-67nnj"] Jul 15 19:33:24.681726 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:24.681696 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-67nnj"] Jul 15 19:33:24.687688 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:24.687633 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:24.720097 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:24.720071 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5adf21e5-0d91-425e-87e9-2114e038e13d-config-file\") pod \"limitador-limitador-64c94fd64f-67nnj\" (UID: \"5adf21e5-0d91-425e-87e9-2114e038e13d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" Jul 15 19:33:24.720250 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:24.720129 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc8xw\" (UniqueName: \"kubernetes.io/projected/5adf21e5-0d91-425e-87e9-2114e038e13d-kube-api-access-xc8xw\") pod \"limitador-limitador-64c94fd64f-67nnj\" (UID: \"5adf21e5-0d91-425e-87e9-2114e038e13d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" Jul 15 19:33:24.739760 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:24.739720 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:24.820846 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:24.820808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5adf21e5-0d91-425e-87e9-2114e038e13d-config-file\") pod \"limitador-limitador-64c94fd64f-67nnj\" (UID: \"5adf21e5-0d91-425e-87e9-2114e038e13d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" Jul 15 19:33:24.821030 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:24.820884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc8xw\" (UniqueName: \"kubernetes.io/projected/5adf21e5-0d91-425e-87e9-2114e038e13d-kube-api-access-xc8xw\") pod \"limitador-limitador-64c94fd64f-67nnj\" (UID: \"5adf21e5-0d91-425e-87e9-2114e038e13d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" Jul 15 19:33:24.821554 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:24.821535 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5adf21e5-0d91-425e-87e9-2114e038e13d-config-file\") pod \"limitador-limitador-64c94fd64f-67nnj\" (UID: \"5adf21e5-0d91-425e-87e9-2114e038e13d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" Jul 15 19:33:24.832301 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:24.832243 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc8xw\" (UniqueName: \"kubernetes.io/projected/5adf21e5-0d91-425e-87e9-2114e038e13d-kube-api-access-xc8xw\") pod \"limitador-limitador-64c94fd64f-67nnj\" (UID: \"5adf21e5-0d91-425e-87e9-2114e038e13d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" Jul 15 19:33:24.897185 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:24.897150 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" Jul 15 19:33:25.041663 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:25.041624 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-67nnj"] Jul 15 19:33:25.043984 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:33:25.043949 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5adf21e5_0d91_425e_87e9_2114e038e13d.slice/crio-e610875684036ad80748e938e44326e6a26c55177c2c580cd9b22b8f936569e8 WatchSource:0}: Error finding container e610875684036ad80748e938e44326e6a26c55177c2c580cd9b22b8f936569e8: Status 404 returned error can't find the container with id e610875684036ad80748e938e44326e6a26c55177c2c580cd9b22b8f936569e8 Jul 15 19:33:25.393501 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:25.393443 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-fn654"] Jul 15 19:33:25.395990 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:25.395968 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-fn654" Jul 15 19:33:25.398177 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:25.398156 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4j5jf\"" Jul 15 19:33:25.404864 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:25.404839 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-fn654"] Jul 15 19:33:25.436136 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:25.436105 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:25.493760 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:25.493694 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:25.526942 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:25.526906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zgn5\" (UniqueName: \"kubernetes.io/projected/31973b97-d9a5-4876-9bf8-1b324075722f-kube-api-access-9zgn5\") pod \"authorino-58cd955675-fn654\" (UID: \"31973b97-d9a5-4876-9bf8-1b324075722f\") " pod="kuadrant-system/authorino-58cd955675-fn654" Jul 15 19:33:25.627549 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:25.627513 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zgn5\" (UniqueName: \"kubernetes.io/projected/31973b97-d9a5-4876-9bf8-1b324075722f-kube-api-access-9zgn5\") pod \"authorino-58cd955675-fn654\" (UID: \"31973b97-d9a5-4876-9bf8-1b324075722f\") " pod="kuadrant-system/authorino-58cd955675-fn654" Jul 15 19:33:25.636299 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:25.636264 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zgn5\" (UniqueName: \"kubernetes.io/projected/31973b97-d9a5-4876-9bf8-1b324075722f-kube-api-access-9zgn5\") pod \"authorino-58cd955675-fn654\" (UID: \"31973b97-d9a5-4876-9bf8-1b324075722f\") " pod="kuadrant-system/authorino-58cd955675-fn654" Jul 15 19:33:25.706288 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:25.706223 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-fn654" Jul 15 19:33:25.754986 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:25.754911 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" event={"ID":"5adf21e5-0d91-425e-87e9-2114e038e13d","Type":"ContainerStarted","Data":"e610875684036ad80748e938e44326e6a26c55177c2c580cd9b22b8f936569e8"} Jul 15 19:33:25.878266 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:25.878229 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-fn654"] Jul 15 19:33:26.763509 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:26.763449 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-fn654" event={"ID":"31973b97-d9a5-4876-9bf8-1b324075722f","Type":"ContainerStarted","Data":"09f15d22898d2db8ee3fb9155ca8a684c385ea7c900f0883dfcea5bd2e7f6a1a"} Jul 15 19:33:29.782749 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:29.782699 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" event={"ID":"5adf21e5-0d91-425e-87e9-2114e038e13d","Type":"ContainerStarted","Data":"4bbdc78071ad9dd098f556d8fa0fe97cfa96aae9dadfbeb9b8b2eddb585b55a1"} Jul 15 19:33:29.783208 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:29.782862 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" Jul 15 19:33:29.784144 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:29.784122 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-fn654" event={"ID":"31973b97-d9a5-4876-9bf8-1b324075722f","Type":"ContainerStarted","Data":"5cc900774feec20ddfe9b19ea1f4cfcb2905e88221bd10a0a416ab424de9d88e"} Jul 15 19:33:29.798624 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:29.798577 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" podStartSLOduration=1.240658522 podStartE2EDuration="5.798561117s" totalImagesPullingTime="4.557902595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:33:29.797599834 +0000 UTC m=+718.269075124" watchObservedRunningTime="2026-07-15 19:33:29.798561117 +0000 UTC m=+718.270036405" Jul 15 19:33:29.813157 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:29.813117 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-fn654" podStartSLOduration=1.087549095 podStartE2EDuration="4.813103742s" totalImagesPullingTime="3.725554647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:33:29.812524003 +0000 UTC m=+718.283999293" watchObservedRunningTime="2026-07-15 19:33:29.813103742 +0000 UTC m=+718.284579039" Jul 15 19:33:30.994498 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:30.994439 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-fn654"] Jul 15 19:33:31.794274 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:31.794236 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-fn654" podUID="31973b97-d9a5-4876-9bf8-1b324075722f" containerName="authorino" containerID="cri-o://5cc900774feec20ddfe9b19ea1f4cfcb2905e88221bd10a0a416ab424de9d88e" gracePeriod=30 Jul 15 19:33:32.050781 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:32.050727 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-fn654" Jul 15 19:33:32.188265 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:32.188232 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zgn5\" (UniqueName: \"kubernetes.io/projected/31973b97-d9a5-4876-9bf8-1b324075722f-kube-api-access-9zgn5\") pod \"31973b97-d9a5-4876-9bf8-1b324075722f\" (UID: \"31973b97-d9a5-4876-9bf8-1b324075722f\") " Jul 15 19:33:32.190349 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:32.190321 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31973b97-d9a5-4876-9bf8-1b324075722f-kube-api-access-9zgn5" (OuterVolumeSpecName: "kube-api-access-9zgn5") pod "31973b97-d9a5-4876-9bf8-1b324075722f" (UID: "31973b97-d9a5-4876-9bf8-1b324075722f"). InnerVolumeSpecName "kube-api-access-9zgn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:33:32.289432 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:32.289394 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zgn5\" (UniqueName: \"kubernetes.io/projected/31973b97-d9a5-4876-9bf8-1b324075722f-kube-api-access-9zgn5\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:33:32.799388 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:32.799352 2564 generic.go:356] "Generic (PLEG): container finished" podID="31973b97-d9a5-4876-9bf8-1b324075722f" containerID="5cc900774feec20ddfe9b19ea1f4cfcb2905e88221bd10a0a416ab424de9d88e" exitCode=0 Jul 15 19:33:32.799388 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:32.799389 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-fn654" Jul 15 19:33:32.799620 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:32.799427 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-fn654" event={"ID":"31973b97-d9a5-4876-9bf8-1b324075722f","Type":"ContainerDied","Data":"5cc900774feec20ddfe9b19ea1f4cfcb2905e88221bd10a0a416ab424de9d88e"} Jul 15 19:33:32.799620 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:32.799460 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-fn654" event={"ID":"31973b97-d9a5-4876-9bf8-1b324075722f","Type":"ContainerDied","Data":"09f15d22898d2db8ee3fb9155ca8a684c385ea7c900f0883dfcea5bd2e7f6a1a"} Jul 15 19:33:32.799620 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:32.799508 2564 scope.go:117] "RemoveContainer" containerID="5cc900774feec20ddfe9b19ea1f4cfcb2905e88221bd10a0a416ab424de9d88e" Jul 15 19:33:32.808437 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:32.808423 2564 scope.go:117] "RemoveContainer" containerID="5cc900774feec20ddfe9b19ea1f4cfcb2905e88221bd10a0a416ab424de9d88e" Jul 15 19:33:32.808770 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:33:32.808750 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc900774feec20ddfe9b19ea1f4cfcb2905e88221bd10a0a416ab424de9d88e\": container with ID starting with 5cc900774feec20ddfe9b19ea1f4cfcb2905e88221bd10a0a416ab424de9d88e not found: ID does not exist" containerID="5cc900774feec20ddfe9b19ea1f4cfcb2905e88221bd10a0a416ab424de9d88e" Jul 15 19:33:32.808844 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:32.808778 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc900774feec20ddfe9b19ea1f4cfcb2905e88221bd10a0a416ab424de9d88e"} err="failed to get container status \"5cc900774feec20ddfe9b19ea1f4cfcb2905e88221bd10a0a416ab424de9d88e\": rpc error: code = NotFound desc = could not find container \"5cc900774feec20ddfe9b19ea1f4cfcb2905e88221bd10a0a416ab424de9d88e\": container with ID starting with 5cc900774feec20ddfe9b19ea1f4cfcb2905e88221bd10a0a416ab424de9d88e not found: ID does not exist" Jul 15 19:33:32.819170 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:32.819145 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-fn654"] Jul 15 19:33:32.820982 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:32.820963 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-fn654"] Jul 15 19:33:34.059736 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:34.059705 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31973b97-d9a5-4876-9bf8-1b324075722f" path="/var/lib/kubelet/pods/31973b97-d9a5-4876-9bf8-1b324075722f/volumes" Jul 15 19:33:40.791196 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:40.791158 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" Jul 15 19:33:41.975598 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:41.975563 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-67nnj"] Jul 15 19:33:41.975968 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:41.975778 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" podUID="5adf21e5-0d91-425e-87e9-2114e038e13d" containerName="limitador" containerID="cri-o://4bbdc78071ad9dd098f556d8fa0fe97cfa96aae9dadfbeb9b8b2eddb585b55a1" gracePeriod=30 Jul 15 19:33:42.525582 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:42.525559 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" Jul 15 19:33:42.677226 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:42.677139 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xc8xw\" (UniqueName: \"kubernetes.io/projected/5adf21e5-0d91-425e-87e9-2114e038e13d-kube-api-access-xc8xw\") pod \"5adf21e5-0d91-425e-87e9-2114e038e13d\" (UID: \"5adf21e5-0d91-425e-87e9-2114e038e13d\") " Jul 15 19:33:42.677377 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:42.677253 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5adf21e5-0d91-425e-87e9-2114e038e13d-config-file\") pod \"5adf21e5-0d91-425e-87e9-2114e038e13d\" (UID: \"5adf21e5-0d91-425e-87e9-2114e038e13d\") " Jul 15 19:33:42.677650 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:42.677631 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5adf21e5-0d91-425e-87e9-2114e038e13d-config-file" (OuterVolumeSpecName: "config-file") pod "5adf21e5-0d91-425e-87e9-2114e038e13d" (UID: "5adf21e5-0d91-425e-87e9-2114e038e13d"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:33:42.679393 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:42.679360 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5adf21e5-0d91-425e-87e9-2114e038e13d-kube-api-access-xc8xw" (OuterVolumeSpecName: "kube-api-access-xc8xw") pod "5adf21e5-0d91-425e-87e9-2114e038e13d" (UID: "5adf21e5-0d91-425e-87e9-2114e038e13d"). InnerVolumeSpecName "kube-api-access-xc8xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:33:42.778128 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:42.778086 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc8xw\" (UniqueName: \"kubernetes.io/projected/5adf21e5-0d91-425e-87e9-2114e038e13d-kube-api-access-xc8xw\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:33:42.778128 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:42.778117 2564 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5adf21e5-0d91-425e-87e9-2114e038e13d-config-file\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:33:42.844274 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:42.844246 2564 generic.go:356] "Generic (PLEG): container finished" podID="5adf21e5-0d91-425e-87e9-2114e038e13d" containerID="4bbdc78071ad9dd098f556d8fa0fe97cfa96aae9dadfbeb9b8b2eddb585b55a1" exitCode=0 Jul 15 19:33:42.844422 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:42.844287 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" event={"ID":"5adf21e5-0d91-425e-87e9-2114e038e13d","Type":"ContainerDied","Data":"4bbdc78071ad9dd098f556d8fa0fe97cfa96aae9dadfbeb9b8b2eddb585b55a1"} Jul 15 19:33:42.844422 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:42.844306 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" Jul 15 19:33:42.844422 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:42.844331 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-67nnj" event={"ID":"5adf21e5-0d91-425e-87e9-2114e038e13d","Type":"ContainerDied","Data":"e610875684036ad80748e938e44326e6a26c55177c2c580cd9b22b8f936569e8"} Jul 15 19:33:42.844422 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:42.844351 2564 scope.go:117] "RemoveContainer" containerID="4bbdc78071ad9dd098f556d8fa0fe97cfa96aae9dadfbeb9b8b2eddb585b55a1" Jul 15 19:33:42.852979 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:42.852960 2564 scope.go:117] "RemoveContainer" containerID="4bbdc78071ad9dd098f556d8fa0fe97cfa96aae9dadfbeb9b8b2eddb585b55a1" Jul 15 19:33:42.853226 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:33:42.853208 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bbdc78071ad9dd098f556d8fa0fe97cfa96aae9dadfbeb9b8b2eddb585b55a1\": container with ID starting with 4bbdc78071ad9dd098f556d8fa0fe97cfa96aae9dadfbeb9b8b2eddb585b55a1 not found: ID does not exist" containerID="4bbdc78071ad9dd098f556d8fa0fe97cfa96aae9dadfbeb9b8b2eddb585b55a1" Jul 15 19:33:42.853276 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:42.853235 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bbdc78071ad9dd098f556d8fa0fe97cfa96aae9dadfbeb9b8b2eddb585b55a1"} err="failed to get container status \"4bbdc78071ad9dd098f556d8fa0fe97cfa96aae9dadfbeb9b8b2eddb585b55a1\": rpc error: code = NotFound desc = could not find container \"4bbdc78071ad9dd098f556d8fa0fe97cfa96aae9dadfbeb9b8b2eddb585b55a1\": container with ID starting with 4bbdc78071ad9dd098f556d8fa0fe97cfa96aae9dadfbeb9b8b2eddb585b55a1 not found: ID does not exist" Jul 15 19:33:42.864917 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:42.864895 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-67nnj"] Jul 15 19:33:42.867004 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:42.866982 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-67nnj"] Jul 15 19:33:44.060381 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:44.060347 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5adf21e5-0d91-425e-87e9-2114e038e13d" path="/var/lib/kubelet/pods/5adf21e5-0d91-425e-87e9-2114e038e13d/volumes" Jul 15 19:33:46.496828 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:46.496787 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:46.596345 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:46.596316 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:46.989149 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:46.989119 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5q7p7"] Jul 15 19:33:46.989456 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:46.989441 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31973b97-d9a5-4876-9bf8-1b324075722f" containerName="authorino" Jul 15 19:33:46.989456 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:46.989455 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="31973b97-d9a5-4876-9bf8-1b324075722f" containerName="authorino" Jul 15 19:33:46.989628 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:46.989489 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5adf21e5-0d91-425e-87e9-2114e038e13d" containerName="limitador" Jul 15 19:33:46.989628 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:46.989495 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5adf21e5-0d91-425e-87e9-2114e038e13d" containerName="limitador" Jul 15 19:33:46.989628 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:46.989565 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5adf21e5-0d91-425e-87e9-2114e038e13d" containerName="limitador" Jul 15 19:33:46.989628 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:46.989573 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="31973b97-d9a5-4876-9bf8-1b324075722f" containerName="authorino" Jul 15 19:33:46.992354 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:46.992333 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q7p7" Jul 15 19:33:46.996829 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:46.996809 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 15 19:33:46.997226 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:46.997213 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 19:33:46.997432 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:46.997419 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 19:33:46.997634 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:46.997619 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-hbcbd\"" Jul 15 19:33:47.009122 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:47.009097 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5q7p7"] Jul 15 19:33:47.069532 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:47.069495 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:47.118804 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:47.118706 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/86420d42-d2e5-4a51-87e2-52d2ef1a7a36-data\") pod \"postgres-69fb4594cf-5q7p7\" (UID: \"86420d42-d2e5-4a51-87e2-52d2ef1a7a36\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q7p7" Jul 15 19:33:47.119008 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:47.118873 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djp2w\" (UniqueName: \"kubernetes.io/projected/86420d42-d2e5-4a51-87e2-52d2ef1a7a36-kube-api-access-djp2w\") pod \"postgres-69fb4594cf-5q7p7\" (UID: \"86420d42-d2e5-4a51-87e2-52d2ef1a7a36\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q7p7" Jul 15 19:33:47.219553 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:47.219514 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/86420d42-d2e5-4a51-87e2-52d2ef1a7a36-data\") pod \"postgres-69fb4594cf-5q7p7\" (UID: \"86420d42-d2e5-4a51-87e2-52d2ef1a7a36\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q7p7" Jul 15 19:33:47.219756 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:47.219570 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djp2w\" (UniqueName: \"kubernetes.io/projected/86420d42-d2e5-4a51-87e2-52d2ef1a7a36-kube-api-access-djp2w\") pod \"postgres-69fb4594cf-5q7p7\" (UID: \"86420d42-d2e5-4a51-87e2-52d2ef1a7a36\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q7p7" Jul 15 19:33:47.219941 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:47.219920 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/86420d42-d2e5-4a51-87e2-52d2ef1a7a36-data\") pod \"postgres-69fb4594cf-5q7p7\" (UID: \"86420d42-d2e5-4a51-87e2-52d2ef1a7a36\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q7p7" Jul 15 19:33:47.231170 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:47.231146 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djp2w\" (UniqueName: \"kubernetes.io/projected/86420d42-d2e5-4a51-87e2-52d2ef1a7a36-kube-api-access-djp2w\") pod \"postgres-69fb4594cf-5q7p7\" (UID: \"86420d42-d2e5-4a51-87e2-52d2ef1a7a36\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q7p7" Jul 15 19:33:47.304314 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:47.304226 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q7p7" Jul 15 19:33:47.443942 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:47.443907 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5q7p7"] Jul 15 19:33:47.447561 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:33:47.447532 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86420d42_d2e5_4a51_87e2_52d2ef1a7a36.slice/crio-bb13e06858666caac18953242a06abb9e523415792e5a7d58853bc4336c65dc8 WatchSource:0}: Error finding container bb13e06858666caac18953242a06abb9e523415792e5a7d58853bc4336c65dc8: Status 404 returned error can't find the container with id bb13e06858666caac18953242a06abb9e523415792e5a7d58853bc4336c65dc8 Jul 15 19:33:47.874046 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:47.874014 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q7p7" event={"ID":"86420d42-d2e5-4a51-87e2-52d2ef1a7a36","Type":"ContainerStarted","Data":"bb13e06858666caac18953242a06abb9e523415792e5a7d58853bc4336c65dc8"} Jul 15 19:33:52.901946 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:52.901916 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q7p7" event={"ID":"86420d42-d2e5-4a51-87e2-52d2ef1a7a36","Type":"ContainerStarted","Data":"7c7310e88954098e55445d67caed19a8c3ea424b2abe68d74172f0c0748aa7f7"} Jul 15 19:33:52.902344 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:52.902026 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q7p7" Jul 15 19:33:52.921456 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:52.921415 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q7p7" podStartSLOduration=2.085998625 podStartE2EDuration="6.921402463s" totalImagesPullingTime="4.835403838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:33:52.919703205 +0000 UTC m=+741.391178494" watchObservedRunningTime="2026-07-15 19:33:52.921402463 +0000 UTC m=+741.392877752" Jul 15 19:33:58.937181 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:58.937152 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q7p7" Jul 15 19:33:59.266529 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.266490 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:59.524981 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.524896 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jbztp"] Jul 15 19:33:59.527293 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.527270 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jbztp" Jul 15 19:33:59.529671 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.529654 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4j5jf\"" Jul 15 19:33:59.536657 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.536633 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jbztp"] Jul 15 19:33:59.626553 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.626520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmj68\" (UniqueName: \"kubernetes.io/projected/851e305b-a863-49c5-b513-9a120526f0b0-kube-api-access-tmj68\") pod \"authorino-5d4b67c446-jbztp\" (UID: \"851e305b-a863-49c5-b513-9a120526f0b0\") " pod="kuadrant-system/authorino-5d4b67c446-jbztp" Jul 15 19:33:59.661293 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.661252 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:59.727016 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.726988 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmj68\" (UniqueName: \"kubernetes.io/projected/851e305b-a863-49c5-b513-9a120526f0b0-kube-api-access-tmj68\") pod \"authorino-5d4b67c446-jbztp\" (UID: \"851e305b-a863-49c5-b513-9a120526f0b0\") " pod="kuadrant-system/authorino-5d4b67c446-jbztp" Jul 15 19:33:59.737074 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.737048 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmj68\" (UniqueName: \"kubernetes.io/projected/851e305b-a863-49c5-b513-9a120526f0b0-kube-api-access-tmj68\") pod \"authorino-5d4b67c446-jbztp\" (UID: \"851e305b-a863-49c5-b513-9a120526f0b0\") " pod="kuadrant-system/authorino-5d4b67c446-jbztp" Jul 15 19:33:59.767172 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.767144 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jbztp"] Jul 15 19:33:59.767393 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.767380 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jbztp" Jul 15 19:33:59.798032 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.798003 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8585d8b49b-dghhj"] Jul 15 19:33:59.800538 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.800516 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8585d8b49b-dghhj" Jul 15 19:33:59.809385 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.809350 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8585d8b49b-dghhj"] Jul 15 19:33:59.828426 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.828384 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnbw8\" (UniqueName: \"kubernetes.io/projected/abc4faea-e9d7-4a16-a0c5-23eafbd03cc3-kube-api-access-rnbw8\") pod \"authorino-8585d8b49b-dghhj\" (UID: \"abc4faea-e9d7-4a16-a0c5-23eafbd03cc3\") " pod="kuadrant-system/authorino-8585d8b49b-dghhj" Jul 15 19:33:59.902333 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.902310 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jbztp"] Jul 15 19:33:59.904758 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:33:59.904732 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod851e305b_a863_49c5_b513_9a120526f0b0.slice/crio-d01efe317910c935a493fe88f7154ebfbf283a5847b30e39a2e12343840fe227 WatchSource:0}: Error finding container d01efe317910c935a493fe88f7154ebfbf283a5847b30e39a2e12343840fe227: Status 404 returned error can't find the container with id d01efe317910c935a493fe88f7154ebfbf283a5847b30e39a2e12343840fe227 Jul 15 19:33:59.928921 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.928901 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnbw8\" (UniqueName: \"kubernetes.io/projected/abc4faea-e9d7-4a16-a0c5-23eafbd03cc3-kube-api-access-rnbw8\") pod \"authorino-8585d8b49b-dghhj\" (UID: \"abc4faea-e9d7-4a16-a0c5-23eafbd03cc3\") " pod="kuadrant-system/authorino-8585d8b49b-dghhj" Jul 15 19:33:59.933459 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.933434 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jbztp" event={"ID":"851e305b-a863-49c5-b513-9a120526f0b0","Type":"ContainerStarted","Data":"d01efe317910c935a493fe88f7154ebfbf283a5847b30e39a2e12343840fe227"} Jul 15 19:33:59.938264 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:33:59.938237 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnbw8\" (UniqueName: \"kubernetes.io/projected/abc4faea-e9d7-4a16-a0c5-23eafbd03cc3-kube-api-access-rnbw8\") pod \"authorino-8585d8b49b-dghhj\" (UID: \"abc4faea-e9d7-4a16-a0c5-23eafbd03cc3\") " pod="kuadrant-system/authorino-8585d8b49b-dghhj" Jul 15 19:34:00.032539 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.032435 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8585d8b49b-dghhj"] Jul 15 19:34:00.032726 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.032711 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8585d8b49b-dghhj" Jul 15 19:34:00.067160 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.067127 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bcdb599d8-qjsks"] Jul 15 19:34:00.069666 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.069643 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bcdb599d8-qjsks" Jul 15 19:34:00.072490 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.072449 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 15 19:34:00.085309 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.085246 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bcdb599d8-qjsks"] Jul 15 19:34:00.130058 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.130029 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06c5580d-f69d-4573-8eca-a30f63b0c511-tls-cert\") pod \"authorino-bcdb599d8-qjsks\" (UID: \"06c5580d-f69d-4573-8eca-a30f63b0c511\") " pod="kuadrant-system/authorino-bcdb599d8-qjsks" Jul 15 19:34:00.130222 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.130073 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgm48\" (UniqueName: \"kubernetes.io/projected/06c5580d-f69d-4573-8eca-a30f63b0c511-kube-api-access-tgm48\") pod \"authorino-bcdb599d8-qjsks\" (UID: \"06c5580d-f69d-4573-8eca-a30f63b0c511\") " pod="kuadrant-system/authorino-bcdb599d8-qjsks" Jul 15 19:34:00.166175 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.165960 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8585d8b49b-dghhj"] Jul 15 19:34:00.168966 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:34:00.168938 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabc4faea_e9d7_4a16_a0c5_23eafbd03cc3.slice/crio-1b95f503ac857d79e74cc6d1ea9865f5b01d72fde9f07773ba9383e370279bf9 WatchSource:0}: Error finding container 1b95f503ac857d79e74cc6d1ea9865f5b01d72fde9f07773ba9383e370279bf9: Status 404 returned error can't find the container with id 1b95f503ac857d79e74cc6d1ea9865f5b01d72fde9f07773ba9383e370279bf9 Jul 15 19:34:00.231540 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.231456 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06c5580d-f69d-4573-8eca-a30f63b0c511-tls-cert\") pod \"authorino-bcdb599d8-qjsks\" (UID: \"06c5580d-f69d-4573-8eca-a30f63b0c511\") " pod="kuadrant-system/authorino-bcdb599d8-qjsks" Jul 15 19:34:00.231713 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.231559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgm48\" (UniqueName: \"kubernetes.io/projected/06c5580d-f69d-4573-8eca-a30f63b0c511-kube-api-access-tgm48\") pod \"authorino-bcdb599d8-qjsks\" (UID: \"06c5580d-f69d-4573-8eca-a30f63b0c511\") " pod="kuadrant-system/authorino-bcdb599d8-qjsks" Jul 15 19:34:00.234057 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.234038 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06c5580d-f69d-4573-8eca-a30f63b0c511-tls-cert\") pod \"authorino-bcdb599d8-qjsks\" (UID: \"06c5580d-f69d-4573-8eca-a30f63b0c511\") " pod="kuadrant-system/authorino-bcdb599d8-qjsks" Jul 15 19:34:00.240098 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.240076 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgm48\" (UniqueName: \"kubernetes.io/projected/06c5580d-f69d-4573-8eca-a30f63b0c511-kube-api-access-tgm48\") pod \"authorino-bcdb599d8-qjsks\" (UID: \"06c5580d-f69d-4573-8eca-a30f63b0c511\") " pod="kuadrant-system/authorino-bcdb599d8-qjsks" Jul 15 19:34:00.380206 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.380125 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bcdb599d8-qjsks" Jul 15 19:34:00.515394 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.515368 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bcdb599d8-qjsks"] Jul 15 19:34:00.519150 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:34:00.519112 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06c5580d_f69d_4573_8eca_a30f63b0c511.slice/crio-793daf6cd25ebf9fb81b2971b79de3bb4638d4343deca1e4edf0f8c92c69d306 WatchSource:0}: Error finding container 793daf6cd25ebf9fb81b2971b79de3bb4638d4343deca1e4edf0f8c92c69d306: Status 404 returned error can't find the container with id 793daf6cd25ebf9fb81b2971b79de3bb4638d4343deca1e4edf0f8c92c69d306 Jul 15 19:34:00.939188 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.939147 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jbztp" event={"ID":"851e305b-a863-49c5-b513-9a120526f0b0","Type":"ContainerStarted","Data":"a5948f946c0e37bc9ae0b7d720389239bfeefd1474851b66a2e0698d67e7561b"} Jul 15 19:34:00.939647 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.939191 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-jbztp" podUID="851e305b-a863-49c5-b513-9a120526f0b0" containerName="authorino" containerID="cri-o://a5948f946c0e37bc9ae0b7d720389239bfeefd1474851b66a2e0698d67e7561b" gracePeriod=30 Jul 15 19:34:00.940674 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.940646 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8585d8b49b-dghhj" event={"ID":"abc4faea-e9d7-4a16-a0c5-23eafbd03cc3","Type":"ContainerStarted","Data":"1b95f503ac857d79e74cc6d1ea9865f5b01d72fde9f07773ba9383e370279bf9"} Jul 15 19:34:00.941799 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.941778 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bcdb599d8-qjsks" event={"ID":"06c5580d-f69d-4573-8eca-a30f63b0c511","Type":"ContainerStarted","Data":"793daf6cd25ebf9fb81b2971b79de3bb4638d4343deca1e4edf0f8c92c69d306"} Jul 15 19:34:00.954694 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:00.954654 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-jbztp" podStartSLOduration=1.3832832879999999 podStartE2EDuration="1.954641244s" totalImagesPullingTime="571.357956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:34:00.953176045 +0000 UTC m=+749.424651337" watchObservedRunningTime="2026-07-15 19:34:00.954641244 +0000 UTC m=+749.426116573" Jul 15 19:34:01.191441 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:01.191379 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jbztp" Jul 15 19:34:01.241359 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:01.241324 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmj68\" (UniqueName: \"kubernetes.io/projected/851e305b-a863-49c5-b513-9a120526f0b0-kube-api-access-tmj68\") pod \"851e305b-a863-49c5-b513-9a120526f0b0\" (UID: \"851e305b-a863-49c5-b513-9a120526f0b0\") " Jul 15 19:34:01.243608 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:01.243580 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851e305b-a863-49c5-b513-9a120526f0b0-kube-api-access-tmj68" (OuterVolumeSpecName: "kube-api-access-tmj68") pod "851e305b-a863-49c5-b513-9a120526f0b0" (UID: "851e305b-a863-49c5-b513-9a120526f0b0"). InnerVolumeSpecName "kube-api-access-tmj68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:34:01.342491 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:01.342439 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmj68\" (UniqueName: \"kubernetes.io/projected/851e305b-a863-49c5-b513-9a120526f0b0-kube-api-access-tmj68\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:34:01.947409 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:01.947321 2564 generic.go:356] "Generic (PLEG): container finished" podID="851e305b-a863-49c5-b513-9a120526f0b0" containerID="a5948f946c0e37bc9ae0b7d720389239bfeefd1474851b66a2e0698d67e7561b" exitCode=0 Jul 15 19:34:01.947409 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:01.947361 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jbztp" Jul 15 19:34:01.947913 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:01.947413 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jbztp" event={"ID":"851e305b-a863-49c5-b513-9a120526f0b0","Type":"ContainerDied","Data":"a5948f946c0e37bc9ae0b7d720389239bfeefd1474851b66a2e0698d67e7561b"} Jul 15 19:34:01.947913 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:01.947453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jbztp" event={"ID":"851e305b-a863-49c5-b513-9a120526f0b0","Type":"ContainerDied","Data":"d01efe317910c935a493fe88f7154ebfbf283a5847b30e39a2e12343840fe227"} Jul 15 19:34:01.947913 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:01.947491 2564 scope.go:117] "RemoveContainer" containerID="a5948f946c0e37bc9ae0b7d720389239bfeefd1474851b66a2e0698d67e7561b" Jul 15 19:34:01.949154 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:01.949125 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8585d8b49b-dghhj" event={"ID":"abc4faea-e9d7-4a16-a0c5-23eafbd03cc3","Type":"ContainerStarted","Data":"013adbaa5af1948b880aaebea9cb49b961cb88058c38a5c8e3a928fde3a9807c"} Jul 15 19:34:01.949535 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:01.949152 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8585d8b49b-dghhj" podUID="abc4faea-e9d7-4a16-a0c5-23eafbd03cc3" containerName="authorino" containerID="cri-o://013adbaa5af1948b880aaebea9cb49b961cb88058c38a5c8e3a928fde3a9807c" gracePeriod=30 Jul 15 19:34:01.950857 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:01.950832 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bcdb599d8-qjsks" event={"ID":"06c5580d-f69d-4573-8eca-a30f63b0c511","Type":"ContainerStarted","Data":"245b59f06f81b026cff2c5bd23df7261f66d08a618c2b246f741c5140202b854"} Jul 15 19:34:01.956947 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:01.956912 2564 scope.go:117] "RemoveContainer" containerID="a5948f946c0e37bc9ae0b7d720389239bfeefd1474851b66a2e0698d67e7561b" Jul 15 19:34:01.957245 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:34:01.957225 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5948f946c0e37bc9ae0b7d720389239bfeefd1474851b66a2e0698d67e7561b\": container with ID starting with a5948f946c0e37bc9ae0b7d720389239bfeefd1474851b66a2e0698d67e7561b not found: ID does not exist" containerID="a5948f946c0e37bc9ae0b7d720389239bfeefd1474851b66a2e0698d67e7561b" Jul 15 19:34:01.957315 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:01.957252 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5948f946c0e37bc9ae0b7d720389239bfeefd1474851b66a2e0698d67e7561b"} err="failed to get container status \"a5948f946c0e37bc9ae0b7d720389239bfeefd1474851b66a2e0698d67e7561b\": rpc error: code = NotFound desc = could not find container \"a5948f946c0e37bc9ae0b7d720389239bfeefd1474851b66a2e0698d67e7561b\": container with ID starting with a5948f946c0e37bc9ae0b7d720389239bfeefd1474851b66a2e0698d67e7561b not found: ID does not exist" Jul 15 19:34:02.000125 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:02.000084 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8585d8b49b-dghhj" podStartSLOduration=1.807316829 podStartE2EDuration="3.000071804s" totalImagesPullingTime="1.192754975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:34:01.98074841 +0000 UTC m=+750.452223698" watchObservedRunningTime="2026-07-15 19:34:02.000071804 +0000 UTC m=+750.471547115" Jul 15 19:34:02.000319 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:02.000302 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bcdb599d8-qjsks" podStartSLOduration=0.863577919 podStartE2EDuration="2.000297557s" totalImagesPullingTime="1.136719638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:34:01.999146103 +0000 UTC m=+750.470621403" watchObservedRunningTime="2026-07-15 19:34:02.000297557 +0000 UTC m=+750.471772892" Jul 15 19:34:02.046827 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:02.046793 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jbztp"] Jul 15 19:34:02.061709 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:02.061634 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jbztp"] Jul 15 19:34:02.207362 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:02.207299 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8585d8b49b-dghhj" Jul 15 19:34:02.251945 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:02.251919 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnbw8\" (UniqueName: \"kubernetes.io/projected/abc4faea-e9d7-4a16-a0c5-23eafbd03cc3-kube-api-access-rnbw8\") pod \"abc4faea-e9d7-4a16-a0c5-23eafbd03cc3\" (UID: \"abc4faea-e9d7-4a16-a0c5-23eafbd03cc3\") " Jul 15 19:34:02.254281 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:02.254250 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc4faea-e9d7-4a16-a0c5-23eafbd03cc3-kube-api-access-rnbw8" (OuterVolumeSpecName: "kube-api-access-rnbw8") pod "abc4faea-e9d7-4a16-a0c5-23eafbd03cc3" (UID: "abc4faea-e9d7-4a16-a0c5-23eafbd03cc3"). InnerVolumeSpecName "kube-api-access-rnbw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:34:02.352435 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:02.352406 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnbw8\" (UniqueName: \"kubernetes.io/projected/abc4faea-e9d7-4a16-a0c5-23eafbd03cc3-kube-api-access-rnbw8\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:34:02.956527 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:02.956497 2564 generic.go:356] "Generic (PLEG): container finished" podID="abc4faea-e9d7-4a16-a0c5-23eafbd03cc3" containerID="013adbaa5af1948b880aaebea9cb49b961cb88058c38a5c8e3a928fde3a9807c" exitCode=0 Jul 15 19:34:02.956527 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:02.956527 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8585d8b49b-dghhj" Jul 15 19:34:02.956946 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:02.956583 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8585d8b49b-dghhj" event={"ID":"abc4faea-e9d7-4a16-a0c5-23eafbd03cc3","Type":"ContainerDied","Data":"013adbaa5af1948b880aaebea9cb49b961cb88058c38a5c8e3a928fde3a9807c"} Jul 15 19:34:02.956946 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:02.956805 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8585d8b49b-dghhj" event={"ID":"abc4faea-e9d7-4a16-a0c5-23eafbd03cc3","Type":"ContainerDied","Data":"1b95f503ac857d79e74cc6d1ea9865f5b01d72fde9f07773ba9383e370279bf9"} Jul 15 19:34:02.957047 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:02.956998 2564 scope.go:117] "RemoveContainer" containerID="013adbaa5af1948b880aaebea9cb49b961cb88058c38a5c8e3a928fde3a9807c" Jul 15 19:34:02.971681 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:02.971663 2564 scope.go:117] "RemoveContainer" containerID="013adbaa5af1948b880aaebea9cb49b961cb88058c38a5c8e3a928fde3a9807c" Jul 15 19:34:02.971971 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:34:02.971954 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"013adbaa5af1948b880aaebea9cb49b961cb88058c38a5c8e3a928fde3a9807c\": container with ID starting with 013adbaa5af1948b880aaebea9cb49b961cb88058c38a5c8e3a928fde3a9807c not found: ID does not exist" containerID="013adbaa5af1948b880aaebea9cb49b961cb88058c38a5c8e3a928fde3a9807c" Jul 15 19:34:02.972034 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:02.971979 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013adbaa5af1948b880aaebea9cb49b961cb88058c38a5c8e3a928fde3a9807c"} err="failed to get container status \"013adbaa5af1948b880aaebea9cb49b961cb88058c38a5c8e3a928fde3a9807c\": rpc error: code = NotFound desc = could not find container \"013adbaa5af1948b880aaebea9cb49b961cb88058c38a5c8e3a928fde3a9807c\": container with ID starting with 013adbaa5af1948b880aaebea9cb49b961cb88058c38a5c8e3a928fde3a9807c not found: ID does not exist" Jul 15 19:34:02.980996 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:02.980972 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8585d8b49b-dghhj"] Jul 15 19:34:02.988826 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:02.988806 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8585d8b49b-dghhj"] Jul 15 19:34:04.059283 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:04.059247 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="851e305b-a863-49c5-b513-9a120526f0b0" path="/var/lib/kubelet/pods/851e305b-a863-49c5-b513-9a120526f0b0/volumes" Jul 15 19:34:04.059668 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:04.059563 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abc4faea-e9d7-4a16-a0c5-23eafbd03cc3" path="/var/lib/kubelet/pods/abc4faea-e9d7-4a16-a0c5-23eafbd03cc3/volumes" Jul 15 19:34:07.111018 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.110966 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:34:07.171307 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.171268 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-767bd6f858-62894"] Jul 15 19:34:07.171654 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.171640 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc4faea-e9d7-4a16-a0c5-23eafbd03cc3" containerName="authorino" Jul 15 19:34:07.171718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.171657 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc4faea-e9d7-4a16-a0c5-23eafbd03cc3" containerName="authorino" Jul 15 19:34:07.171718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.171673 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851e305b-a863-49c5-b513-9a120526f0b0" containerName="authorino" Jul 15 19:34:07.171718 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.171681 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="851e305b-a863-49c5-b513-9a120526f0b0" containerName="authorino" Jul 15 19:34:07.171808 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.171747 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc4faea-e9d7-4a16-a0c5-23eafbd03cc3" containerName="authorino" Jul 15 19:34:07.171808 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.171759 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="851e305b-a863-49c5-b513-9a120526f0b0" containerName="authorino" Jul 15 19:34:07.174420 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.174396 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-767bd6f858-62894" Jul 15 19:34:07.180121 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:34:07.180094 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" type="*v1.Secret" Jul 15 19:34:07.180121 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:34:07.180094 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-controller-dockercfg-8c6qr\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8c6qr\"" type="*v1.Secret" Jul 15 19:34:07.180722 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:34:07.180692 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-controller-767bd6f858-62894\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" podUID="e78230f7-9e8d-4ee3-b76d-9d62abe80983" pod="opendatahub/maas-controller-767bd6f858-62894" Jul 15 19:34:07.180863 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:34:07.180778 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"maas-parameters\" is forbidden: User \"system:node:ip-10-0-141-225.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-225.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"maas-parameters\"" type="*v1.ConfigMap" Jul 15 19:34:07.201872 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.201841 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-767bd6f858-62894"] Jul 15 19:34:07.290551 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.290518 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:34:07.292742 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.292725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk6bg\" (UniqueName: \"kubernetes.io/projected/e78230f7-9e8d-4ee3-b76d-9d62abe80983-kube-api-access-wk6bg\") pod \"maas-controller-767bd6f858-62894\" (UID: \"e78230f7-9e8d-4ee3-b76d-9d62abe80983\") " pod="opendatahub/maas-controller-767bd6f858-62894" Jul 15 19:34:07.292840 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.292761 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e78230f7-9e8d-4ee3-b76d-9d62abe80983-webhook-cert\") pod \"maas-controller-767bd6f858-62894\" (UID: \"e78230f7-9e8d-4ee3-b76d-9d62abe80983\") " pod="opendatahub/maas-controller-767bd6f858-62894" Jul 15 19:34:07.365562 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.365454 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:34:07.393728 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.393696 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e78230f7-9e8d-4ee3-b76d-9d62abe80983-webhook-cert\") pod \"maas-controller-767bd6f858-62894\" (UID: \"e78230f7-9e8d-4ee3-b76d-9d62abe80983\") " pod="opendatahub/maas-controller-767bd6f858-62894" Jul 15 19:34:07.393881 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.393781 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk6bg\" (UniqueName: \"kubernetes.io/projected/e78230f7-9e8d-4ee3-b76d-9d62abe80983-kube-api-access-wk6bg\") pod \"maas-controller-767bd6f858-62894\" (UID: \"e78230f7-9e8d-4ee3-b76d-9d62abe80983\") " pod="opendatahub/maas-controller-767bd6f858-62894" Jul 15 19:34:07.406079 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.406048 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk6bg\" (UniqueName: \"kubernetes.io/projected/e78230f7-9e8d-4ee3-b76d-9d62abe80983-kube-api-access-wk6bg\") pod \"maas-controller-767bd6f858-62894\" (UID: \"e78230f7-9e8d-4ee3-b76d-9d62abe80983\") " pod="opendatahub/maas-controller-767bd6f858-62894" Jul 15 19:34:07.441196 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.441163 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6548747c7b-t8sd9"] Jul 15 19:34:07.443409 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.443395 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6548747c7b-t8sd9" Jul 15 19:34:07.464198 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.464174 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6548747c7b-t8sd9"] Jul 15 19:34:07.494304 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.494279 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2-webhook-cert\") pod \"maas-controller-6548747c7b-t8sd9\" (UID: \"9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2\") " pod="opendatahub/maas-controller-6548747c7b-t8sd9" Jul 15 19:34:07.494443 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.494316 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crfld\" (UniqueName: \"kubernetes.io/projected/9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2-kube-api-access-crfld\") pod \"maas-controller-6548747c7b-t8sd9\" (UID: \"9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2\") " pod="opendatahub/maas-controller-6548747c7b-t8sd9" Jul 15 19:34:07.595441 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.595404 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2-webhook-cert\") pod \"maas-controller-6548747c7b-t8sd9\" (UID: \"9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2\") " pod="opendatahub/maas-controller-6548747c7b-t8sd9" Jul 15 19:34:07.595441 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.595444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crfld\" (UniqueName: \"kubernetes.io/projected/9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2-kube-api-access-crfld\") pod \"maas-controller-6548747c7b-t8sd9\" (UID: \"9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2\") " pod="opendatahub/maas-controller-6548747c7b-t8sd9" Jul 15 19:34:07.609740 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:07.609704 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crfld\" (UniqueName: \"kubernetes.io/projected/9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2-kube-api-access-crfld\") pod \"maas-controller-6548747c7b-t8sd9\" (UID: \"9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2\") " pod="opendatahub/maas-controller-6548747c7b-t8sd9" Jul 15 19:34:08.078267 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:08.078234 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8c6qr\"" Jul 15 19:34:08.222014 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:08.221978 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 15 19:34:08.226559 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:08.226536 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e78230f7-9e8d-4ee3-b76d-9d62abe80983-webhook-cert\") pod \"maas-controller-767bd6f858-62894\" (UID: \"e78230f7-9e8d-4ee3-b76d-9d62abe80983\") " pod="opendatahub/maas-controller-767bd6f858-62894" Jul 15 19:34:08.228259 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:08.228237 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2-webhook-cert\") pod \"maas-controller-6548747c7b-t8sd9\" (UID: \"9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2\") " pod="opendatahub/maas-controller-6548747c7b-t8sd9" Jul 15 19:34:08.353355 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:08.353271 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6548747c7b-t8sd9" Jul 15 19:34:08.386727 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:08.386689 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-767bd6f858-62894" Jul 15 19:34:08.510741 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:08.510696 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6548747c7b-t8sd9"] Jul 15 19:34:08.513961 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:34:08.513915 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b89e3c2_7ee4_46ed_8b51_6812a8c3bce2.slice/crio-4ca686ce8344b76ffbe2c519c456c64469c76e24c8e87789b4115fdff06e289f WatchSource:0}: Error finding container 4ca686ce8344b76ffbe2c519c456c64469c76e24c8e87789b4115fdff06e289f: Status 404 returned error can't find the container with id 4ca686ce8344b76ffbe2c519c456c64469c76e24c8e87789b4115fdff06e289f Jul 15 19:34:08.550086 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:08.550066 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-767bd6f858-62894"] Jul 15 19:34:08.552572 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:34:08.552540 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78230f7_9e8d_4ee3_b76d_9d62abe80983.slice/crio-6afbc6463b4a6351db4b279c997372f4073cd3c168da492016a66995a467deb2 WatchSource:0}: Error finding container 6afbc6463b4a6351db4b279c997372f4073cd3c168da492016a66995a467deb2: Status 404 returned error can't find the container with id 6afbc6463b4a6351db4b279c997372f4073cd3c168da492016a66995a467deb2 Jul 15 19:34:08.700968 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:08.700880 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 15 19:34:08.985485 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:08.985444 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-767bd6f858-62894" event={"ID":"e78230f7-9e8d-4ee3-b76d-9d62abe80983","Type":"ContainerStarted","Data":"6afbc6463b4a6351db4b279c997372f4073cd3c168da492016a66995a467deb2"} Jul 15 19:34:08.986591 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:08.986564 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6548747c7b-t8sd9" event={"ID":"9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2","Type":"ContainerStarted","Data":"4ca686ce8344b76ffbe2c519c456c64469c76e24c8e87789b4115fdff06e289f"} Jul 15 19:34:11.595891 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:11.595802 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:34:11.694568 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:11.694535 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:34:12.004962 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:12.004923 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-767bd6f858-62894" event={"ID":"e78230f7-9e8d-4ee3-b76d-9d62abe80983","Type":"ContainerStarted","Data":"885d3d7e29c1a145fd80b2f073238f23e0fa5d3bb9c42feaa16c04db8855877b"} Jul 15 19:34:12.005163 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:12.004996 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-767bd6f858-62894" Jul 15 19:34:12.006418 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:12.006392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6548747c7b-t8sd9" event={"ID":"9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2","Type":"ContainerStarted","Data":"39c2a178e348d925cffb3f3b8e81aedec4d9bb2af06f258a69f21d9c97ea97d5"} Jul 15 19:34:12.006562 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:12.006436 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6548747c7b-t8sd9" Jul 15 19:34:12.051108 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:12.051031 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-767bd6f858-62894" podStartSLOduration=2.3233646439999998 podStartE2EDuration="5.051011725s" totalImagesPullingTime="2.727647081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:34:12.04853849 +0000 UTC m=+760.520013780" watchObservedRunningTime="2026-07-15 19:34:12.051011725 +0000 UTC m=+760.522487013" Jul 15 19:34:12.093394 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:12.093347 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6548747c7b-t8sd9" podStartSLOduration=2.331654871 podStartE2EDuration="5.093332336s" totalImagesPullingTime="2.761677465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:34:12.090730658 +0000 UTC m=+760.562205929" watchObservedRunningTime="2026-07-15 19:34:12.093332336 +0000 UTC m=+760.564807624" Jul 15 19:34:23.014219 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:23.014192 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6548747c7b-t8sd9" Jul 15 19:34:23.014622 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:23.014265 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-767bd6f858-62894" Jul 15 19:34:23.091390 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:23.091361 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-767bd6f858-62894"] Jul 15 19:34:23.091634 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:23.091568 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-767bd6f858-62894" podUID="e78230f7-9e8d-4ee3-b76d-9d62abe80983" containerName="manager" containerID="cri-o://885d3d7e29c1a145fd80b2f073238f23e0fa5d3bb9c42feaa16c04db8855877b" gracePeriod=30 Jul 15 19:34:23.881669 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:23.881642 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-767bd6f858-62894" Jul 15 19:34:23.937141 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:23.937116 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e78230f7-9e8d-4ee3-b76d-9d62abe80983-webhook-cert\") pod \"e78230f7-9e8d-4ee3-b76d-9d62abe80983\" (UID: \"e78230f7-9e8d-4ee3-b76d-9d62abe80983\") " Jul 15 19:34:23.937294 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:23.937191 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk6bg\" (UniqueName: \"kubernetes.io/projected/e78230f7-9e8d-4ee3-b76d-9d62abe80983-kube-api-access-wk6bg\") pod \"e78230f7-9e8d-4ee3-b76d-9d62abe80983\" (UID: \"e78230f7-9e8d-4ee3-b76d-9d62abe80983\") " Jul 15 19:34:23.939278 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:23.939252 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78230f7-9e8d-4ee3-b76d-9d62abe80983-kube-api-access-wk6bg" (OuterVolumeSpecName: "kube-api-access-wk6bg") pod "e78230f7-9e8d-4ee3-b76d-9d62abe80983" (UID: "e78230f7-9e8d-4ee3-b76d-9d62abe80983"). InnerVolumeSpecName "kube-api-access-wk6bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:34:23.939396 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:23.939271 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78230f7-9e8d-4ee3-b76d-9d62abe80983-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e78230f7-9e8d-4ee3-b76d-9d62abe80983" (UID: "e78230f7-9e8d-4ee3-b76d-9d62abe80983"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:34:24.038228 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:24.038144 2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e78230f7-9e8d-4ee3-b76d-9d62abe80983-webhook-cert\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:34:24.038228 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:24.038176 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk6bg\" (UniqueName: \"kubernetes.io/projected/e78230f7-9e8d-4ee3-b76d-9d62abe80983-kube-api-access-wk6bg\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:34:24.063362 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:24.063330 2564 generic.go:356] "Generic (PLEG): container finished" podID="e78230f7-9e8d-4ee3-b76d-9d62abe80983" containerID="885d3d7e29c1a145fd80b2f073238f23e0fa5d3bb9c42feaa16c04db8855877b" exitCode=0 Jul 15 19:34:24.063362 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:24.063361 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-767bd6f858-62894" Jul 15 19:34:24.063606 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:24.063416 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-767bd6f858-62894" event={"ID":"e78230f7-9e8d-4ee3-b76d-9d62abe80983","Type":"ContainerDied","Data":"885d3d7e29c1a145fd80b2f073238f23e0fa5d3bb9c42feaa16c04db8855877b"} Jul 15 19:34:24.063606 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:24.063465 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-767bd6f858-62894" event={"ID":"e78230f7-9e8d-4ee3-b76d-9d62abe80983","Type":"ContainerDied","Data":"6afbc6463b4a6351db4b279c997372f4073cd3c168da492016a66995a467deb2"} Jul 15 19:34:24.063606 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:24.063506 2564 scope.go:117] "RemoveContainer" containerID="885d3d7e29c1a145fd80b2f073238f23e0fa5d3bb9c42feaa16c04db8855877b" Jul 15 19:34:24.073291 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:24.073275 2564 scope.go:117] "RemoveContainer" containerID="885d3d7e29c1a145fd80b2f073238f23e0fa5d3bb9c42feaa16c04db8855877b" Jul 15 19:34:24.073587 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:34:24.073571 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885d3d7e29c1a145fd80b2f073238f23e0fa5d3bb9c42feaa16c04db8855877b\": container with ID starting with 885d3d7e29c1a145fd80b2f073238f23e0fa5d3bb9c42feaa16c04db8855877b not found: ID does not exist" containerID="885d3d7e29c1a145fd80b2f073238f23e0fa5d3bb9c42feaa16c04db8855877b" Jul 15 19:34:24.073632 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:24.073595 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885d3d7e29c1a145fd80b2f073238f23e0fa5d3bb9c42feaa16c04db8855877b"} err="failed to get container status \"885d3d7e29c1a145fd80b2f073238f23e0fa5d3bb9c42feaa16c04db8855877b\": rpc error: code = NotFound desc = could not find container \"885d3d7e29c1a145fd80b2f073238f23e0fa5d3bb9c42feaa16c04db8855877b\": container with ID starting with 885d3d7e29c1a145fd80b2f073238f23e0fa5d3bb9c42feaa16c04db8855877b not found: ID does not exist" Jul 15 19:34:24.084678 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:24.084655 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-767bd6f858-62894"] Jul 15 19:34:24.088336 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:24.088315 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-767bd6f858-62894"] Jul 15 19:34:26.062090 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:26.062050 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e78230f7-9e8d-4ee3-b76d-9d62abe80983" path="/var/lib/kubelet/pods/e78230f7-9e8d-4ee3-b76d-9d62abe80983/volumes" Jul 15 19:34:42.346128 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:42.346099 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:34:43.484801 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:43.484770 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:34:43.578637 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:43.578601 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:34:43.977811 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:43.977784 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:34:44.087043 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.087015 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86bc485959-qjbnl"] Jul 15 19:34:44.087358 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.087346 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78230f7-9e8d-4ee3-b76d-9d62abe80983" containerName="manager" Jul 15 19:34:44.087403 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.087359 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78230f7-9e8d-4ee3-b76d-9d62abe80983" containerName="manager" Jul 15 19:34:44.087441 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.087429 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e78230f7-9e8d-4ee3-b76d-9d62abe80983" containerName="manager" Jul 15 19:34:44.091736 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.091711 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86bc485959-qjbnl" Jul 15 19:34:44.094747 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.094722 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7cwr8\"" Jul 15 19:34:44.094882 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.094746 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 15 19:34:44.106408 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.106383 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86bc485959-qjbnl"] Jul 15 19:34:44.216011 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.215981 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c873091d-6ab8-4c72-85ab-f5bd0ddc5365-maas-api-tls\") pod \"maas-api-86bc485959-qjbnl\" (UID: \"c873091d-6ab8-4c72-85ab-f5bd0ddc5365\") " pod="odh-ai-gateway-infra/maas-api-86bc485959-qjbnl" Jul 15 19:34:44.216229 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.216081 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnpnv\" (UniqueName: \"kubernetes.io/projected/c873091d-6ab8-4c72-85ab-f5bd0ddc5365-kube-api-access-rnpnv\") pod \"maas-api-86bc485959-qjbnl\" (UID: \"c873091d-6ab8-4c72-85ab-f5bd0ddc5365\") " pod="odh-ai-gateway-infra/maas-api-86bc485959-qjbnl" Jul 15 19:34:44.216229 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.216111 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c873091d-6ab8-4c72-85ab-f5bd0ddc5365-openshift-service-ca\") pod \"maas-api-86bc485959-qjbnl\" (UID: \"c873091d-6ab8-4c72-85ab-f5bd0ddc5365\") " pod="odh-ai-gateway-infra/maas-api-86bc485959-qjbnl" Jul 15 19:34:44.315033 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.314924 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:34:44.317052 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.317024 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnpnv\" (UniqueName: \"kubernetes.io/projected/c873091d-6ab8-4c72-85ab-f5bd0ddc5365-kube-api-access-rnpnv\") pod \"maas-api-86bc485959-qjbnl\" (UID: \"c873091d-6ab8-4c72-85ab-f5bd0ddc5365\") " pod="odh-ai-gateway-infra/maas-api-86bc485959-qjbnl" Jul 15 19:34:44.317195 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.317057 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c873091d-6ab8-4c72-85ab-f5bd0ddc5365-openshift-service-ca\") pod \"maas-api-86bc485959-qjbnl\" (UID: \"c873091d-6ab8-4c72-85ab-f5bd0ddc5365\") " pod="odh-ai-gateway-infra/maas-api-86bc485959-qjbnl" Jul 15 19:34:44.317195 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.317087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c873091d-6ab8-4c72-85ab-f5bd0ddc5365-maas-api-tls\") pod \"maas-api-86bc485959-qjbnl\" (UID: \"c873091d-6ab8-4c72-85ab-f5bd0ddc5365\") " pod="odh-ai-gateway-infra/maas-api-86bc485959-qjbnl" Jul 15 19:34:44.317766 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.317740 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c873091d-6ab8-4c72-85ab-f5bd0ddc5365-openshift-service-ca\") pod \"maas-api-86bc485959-qjbnl\" (UID: \"c873091d-6ab8-4c72-85ab-f5bd0ddc5365\") " pod="odh-ai-gateway-infra/maas-api-86bc485959-qjbnl" Jul 15 19:34:44.319544 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.319522 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c873091d-6ab8-4c72-85ab-f5bd0ddc5365-maas-api-tls\") pod \"maas-api-86bc485959-qjbnl\" (UID: \"c873091d-6ab8-4c72-85ab-f5bd0ddc5365\") " pod="odh-ai-gateway-infra/maas-api-86bc485959-qjbnl" Jul 15 19:34:44.330719 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.330687 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnpnv\" (UniqueName: \"kubernetes.io/projected/c873091d-6ab8-4c72-85ab-f5bd0ddc5365-kube-api-access-rnpnv\") pod \"maas-api-86bc485959-qjbnl\" (UID: \"c873091d-6ab8-4c72-85ab-f5bd0ddc5365\") " pod="odh-ai-gateway-infra/maas-api-86bc485959-qjbnl" Jul 15 19:34:44.402989 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.402948 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86bc485959-qjbnl" Jul 15 19:34:44.461739 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.461705 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:34:44.555844 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.555815 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86bc485959-qjbnl"] Jul 15 19:34:44.558830 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:34:44.558800 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc873091d_6ab8_4c72_85ab_f5bd0ddc5365.slice/crio-3abb63ab45bd3f6f414aea0106ed18ebd1dae319ef6493454febe2a67eeabb41 WatchSource:0}: Error finding container 3abb63ab45bd3f6f414aea0106ed18ebd1dae319ef6493454febe2a67eeabb41: Status 404 returned error can't find the container with id 3abb63ab45bd3f6f414aea0106ed18ebd1dae319ef6493454febe2a67eeabb41 Jul 15 19:34:44.565395 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.565341 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:34:44.635560 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.635523 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:34:44.637193 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.637167 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-7gkbp"] Jul 15 19:34:44.641775 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.641757 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-7gkbp" Jul 15 19:34:44.644801 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.644680 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 15 19:34:44.653946 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.653924 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-7gkbp"] Jul 15 19:34:44.822554 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.822483 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmjw4\" (UniqueName: \"kubernetes.io/projected/a3701cc2-6375-4df3-99f4-f15804d6e560-kube-api-access-tmjw4\") pod \"payload-pre-processing-74584bb9f8-7gkbp\" (UID: \"a3701cc2-6375-4df3-99f4-f15804d6e560\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-7gkbp" Jul 15 19:34:44.822726 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.822635 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3701cc2-6375-4df3-99f4-f15804d6e560-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-7gkbp\" (UID: \"a3701cc2-6375-4df3-99f4-f15804d6e560\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-7gkbp" Jul 15 19:34:44.923161 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.923126 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3701cc2-6375-4df3-99f4-f15804d6e560-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-7gkbp\" (UID: \"a3701cc2-6375-4df3-99f4-f15804d6e560\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-7gkbp" Jul 15 19:34:44.923312 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.923174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmjw4\" (UniqueName: \"kubernetes.io/projected/a3701cc2-6375-4df3-99f4-f15804d6e560-kube-api-access-tmjw4\") pod \"payload-pre-processing-74584bb9f8-7gkbp\" (UID: \"a3701cc2-6375-4df3-99f4-f15804d6e560\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-7gkbp" Jul 15 19:34:44.923823 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.923802 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3701cc2-6375-4df3-99f4-f15804d6e560-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-7gkbp\" (UID: \"a3701cc2-6375-4df3-99f4-f15804d6e560\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-7gkbp" Jul 15 19:34:44.932259 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.932236 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmjw4\" (UniqueName: \"kubernetes.io/projected/a3701cc2-6375-4df3-99f4-f15804d6e560-kube-api-access-tmjw4\") pod \"payload-pre-processing-74584bb9f8-7gkbp\" (UID: \"a3701cc2-6375-4df3-99f4-f15804d6e560\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-7gkbp" Jul 15 19:34:44.954192 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:44.954163 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-7gkbp" Jul 15 19:34:45.098158 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:45.098082 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-7gkbp"] Jul 15 19:34:45.101443 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:34:45.101421 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3701cc2_6375_4df3_99f4_f15804d6e560.slice/crio-34aa464f52746626a30fdeb137a6500be78c33826407acda9b78f5f6d908271b WatchSource:0}: Error finding container 34aa464f52746626a30fdeb137a6500be78c33826407acda9b78f5f6d908271b: Status 404 returned error can't find the container with id 34aa464f52746626a30fdeb137a6500be78c33826407acda9b78f5f6d908271b Jul 15 19:34:45.163006 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:45.162972 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86bc485959-qjbnl" event={"ID":"c873091d-6ab8-4c72-85ab-f5bd0ddc5365","Type":"ContainerStarted","Data":"3abb63ab45bd3f6f414aea0106ed18ebd1dae319ef6493454febe2a67eeabb41"} Jul 15 19:34:45.164434 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:45.164393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-7gkbp" event={"ID":"a3701cc2-6375-4df3-99f4-f15804d6e560","Type":"ContainerStarted","Data":"34aa464f52746626a30fdeb137a6500be78c33826407acda9b78f5f6d908271b"} Jul 15 19:34:47.176841 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:47.176744 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-7gkbp" event={"ID":"a3701cc2-6375-4df3-99f4-f15804d6e560","Type":"ContainerStarted","Data":"bd072f9f20a401552d2a21e686d1b6dbb24f32c8b87aa36856dc7b5985638f51"} Jul 15 19:34:47.176841 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:47.176827 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-7gkbp" Jul 15 19:34:47.178224 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:47.178196 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86bc485959-qjbnl" event={"ID":"c873091d-6ab8-4c72-85ab-f5bd0ddc5365","Type":"ContainerStarted","Data":"a040be9a459c294eb1cf4341eb343ac032172e0f7a7219e03af63202af47236e"} Jul 15 19:34:47.178356 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:47.178310 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86bc485959-qjbnl" Jul 15 19:34:47.193552 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:47.193510 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-7gkbp" podStartSLOduration=1.493161632 podStartE2EDuration="3.19349669s" totalImagesPullingTime="1.700335058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:34:47.192913177 +0000 UTC m=+795.664388466" watchObservedRunningTime="2026-07-15 19:34:47.19349669 +0000 UTC m=+795.664971978" Jul 15 19:34:47.211196 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:47.211149 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86bc485959-qjbnl" podStartSLOduration=1.016860094 podStartE2EDuration="3.211133401s" totalImagesPullingTime="2.194273307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:34:47.208951292 +0000 UTC m=+795.680426598" watchObservedRunningTime="2026-07-15 19:34:47.211133401 +0000 UTC m=+795.682608689" Jul 15 19:34:52.203032 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:52.203005 2564 generic.go:356] "Generic (PLEG): container finished" podID="176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd" containerID="a4c5bc43424edb107decb594cedae3f5014bad965ce76b6b694f90ead3e4efa5" exitCode=0 Jul 15 19:34:52.203395 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:52.203064 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" event={"ID":"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd","Type":"ContainerDied","Data":"a4c5bc43424edb107decb594cedae3f5014bad965ce76b6b694f90ead3e4efa5"} Jul 15 19:34:52.203395 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:52.203393 2564 scope.go:117] "RemoveContainer" containerID="a4c5bc43424edb107decb594cedae3f5014bad965ce76b6b694f90ead3e4efa5" Jul 15 19:34:52.204413 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:52.204387 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 19:34:52.204497 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:52.204452 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 19:34:52.204546 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:52.204501 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 19:34:53.188729 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:53.188657 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86bc485959-qjbnl" Jul 15 19:34:53.209535 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:53.209508 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" event={"ID":"176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd","Type":"ContainerStarted","Data":"e8b39bf3f782736acff0c1b512c4e229590102f1ae25a3c038f06085092757f3"} Jul 15 19:34:53.581266 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:53.581227 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:34:53.581521 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:53.581494 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-srxrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 15 19:34:53.581569 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:53.581550 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" podUID="176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 15 19:34:54.580885 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:54.580842 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-srxrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 15 19:34:54.581394 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:54.580926 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" podUID="176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 15 19:34:54.904709 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:54.904634 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:34:54.994102 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:54.994067 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:34:55.581217 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:55.581179 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-srxrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 15 19:34:55.581625 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:55.581232 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" podUID="176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 15 19:34:56.581075 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:56.581035 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-srxrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 15 19:34:56.581251 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:56.581107 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" podUID="176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 15 19:34:57.584868 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:57.584833 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:34:57.585311 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:57.585204 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:34:57.585998 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:57.585980 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-srxrd" Jul 15 19:34:58.186364 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:34:58.186336 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-7gkbp" Jul 15 19:35:03.289416 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.289378 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m"] Jul 15 19:35:03.292264 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.292245 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.295308 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.295287 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 19:35:03.295414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.295311 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-56lms\"" Jul 15 19:35:03.295414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.295287 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 15 19:35:03.295414 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.295294 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 19:35:03.307184 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.307160 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m"] Jul 15 19:35:03.314562 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.314538 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:35:03.399485 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.399440 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd73d95-2a73-43d2-ad0c-6d026ea94931-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.399631 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.399502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fd73d95-2a73-43d2-ad0c-6d026ea94931-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.399631 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.399551 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fd73d95-2a73-43d2-ad0c-6d026ea94931-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.399631 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.399600 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fd73d95-2a73-43d2-ad0c-6d026ea94931-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.399763 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.399642 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fd73d95-2a73-43d2-ad0c-6d026ea94931-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.399763 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.399698 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtxlz\" (UniqueName: \"kubernetes.io/projected/9fd73d95-2a73-43d2-ad0c-6d026ea94931-kube-api-access-xtxlz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.500357 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.500323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fd73d95-2a73-43d2-ad0c-6d026ea94931-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.500565 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.500379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fd73d95-2a73-43d2-ad0c-6d026ea94931-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.500565 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.500397 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fd73d95-2a73-43d2-ad0c-6d026ea94931-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.500565 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.500422 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fd73d95-2a73-43d2-ad0c-6d026ea94931-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.500565 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.500463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtxlz\" (UniqueName: \"kubernetes.io/projected/9fd73d95-2a73-43d2-ad0c-6d026ea94931-kube-api-access-xtxlz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.500565 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.500522 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd73d95-2a73-43d2-ad0c-6d026ea94931-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.500880 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.500858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fd73d95-2a73-43d2-ad0c-6d026ea94931-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.500941 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.500890 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fd73d95-2a73-43d2-ad0c-6d026ea94931-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.500941 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.500923 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fd73d95-2a73-43d2-ad0c-6d026ea94931-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.502797 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.502770 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fd73d95-2a73-43d2-ad0c-6d026ea94931-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.503071 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.503052 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd73d95-2a73-43d2-ad0c-6d026ea94931-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.510210 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.510190 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtxlz\" (UniqueName: \"kubernetes.io/projected/9fd73d95-2a73-43d2-ad0c-6d026ea94931-kube-api-access-xtxlz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m\" (UID: \"9fd73d95-2a73-43d2-ad0c-6d026ea94931\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.602037 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.601956 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:03.742327 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:03.742301 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m"] Jul 15 19:35:03.744841 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:35:03.744812 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd73d95_2a73_43d2_ad0c_6d026ea94931.slice/crio-48036ca06628fc2e7834efbe931b0580e5a438025a1ba779f9803b63954bc1d6 WatchSource:0}: Error finding container 48036ca06628fc2e7834efbe931b0580e5a438025a1ba779f9803b63954bc1d6: Status 404 returned error can't find the container with id 48036ca06628fc2e7834efbe931b0580e5a438025a1ba779f9803b63954bc1d6 Jul 15 19:35:04.271921 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:04.271873 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" event={"ID":"9fd73d95-2a73-43d2-ad0c-6d026ea94931","Type":"ContainerStarted","Data":"48036ca06628fc2e7834efbe931b0580e5a438025a1ba779f9803b63954bc1d6"} Jul 15 19:35:04.627772 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:04.627675 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:35:05.279543 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:05.279506 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" event={"ID":"9fd73d95-2a73-43d2-ad0c-6d026ea94931","Type":"ContainerStarted","Data":"bd2a8a1c6bdcf4850e36dec5c66ada78337f2364fbac9ecb1e9267b9803556ea"} Jul 15 19:35:05.301491 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:05.301396 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" podStartSLOduration=0.876878387 podStartE2EDuration="2.301373725s" totalImagesPullingTime="1.424495338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:35:05.300027015 +0000 UTC m=+813.771502304" watchObservedRunningTime="2026-07-15 19:35:05.301373725 +0000 UTC m=+813.772849016" Jul 15 19:35:13.602284 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:13.602241 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:13.602823 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:13.602300 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:13.614780 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:13.614754 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:14.332561 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:14.332533 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m" Jul 15 19:35:14.605279 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:14.605200 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:35:20.081560 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:20.081515 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:35:28.808877 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:28.808835 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz"] Jul 15 19:35:28.813961 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:28.813936 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:28.816900 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:28.816870 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 15 19:35:28.842079 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:28.842056 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz"] Jul 15 19:35:28.914040 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:28.914001 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:28.914223 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:28.914071 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:28.914223 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:28.914100 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:28.914223 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:28.914171 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxld2\" (UniqueName: \"kubernetes.io/projected/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-kube-api-access-kxld2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:28.914223 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:28.914207 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:28.914446 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:28.914243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:28.920253 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:28.920225 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:35:29.015725 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:29.015683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:29.015898 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:29.015746 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:29.015898 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:29.015806 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxld2\" (UniqueName: \"kubernetes.io/projected/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-kube-api-access-kxld2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:29.015898 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:29.015842 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:29.015898 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:29.015887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:29.016121 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:29.015955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:29.016364 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:29.016336 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:29.016519 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:29.016490 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:29.016796 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:29.016738 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:29.018245 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:29.018221 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:29.018782 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:29.018754 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:29.024122 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:29.024103 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxld2\" (UniqueName: \"kubernetes.io/projected/6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb-kube-api-access-kxld2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz\" (UID: \"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:29.123869 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:29.123777 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:29.269649 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:29.269619 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz"] Jul 15 19:35:29.272659 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:35:29.272622 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a63f9a5_f0f3_4c36_bf79_7b1c94042ffb.slice/crio-a9d19d3111b359020f0c27e45b70cd854389b61c045c49f406f9a2b02f65ed6e WatchSource:0}: Error finding container a9d19d3111b359020f0c27e45b70cd854389b61c045c49f406f9a2b02f65ed6e: Status 404 returned error can't find the container with id a9d19d3111b359020f0c27e45b70cd854389b61c045c49f406f9a2b02f65ed6e Jul 15 19:35:29.394825 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:29.394741 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" event={"ID":"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb","Type":"ContainerStarted","Data":"a9d19d3111b359020f0c27e45b70cd854389b61c045c49f406f9a2b02f65ed6e"} Jul 15 19:35:30.401202 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:30.401164 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" event={"ID":"6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb","Type":"ContainerStarted","Data":"9db09550fcedafbd81a00963f1d785ac217f41fe9d26052e7295900e96194a6f"} Jul 15 19:35:30.420578 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:30.420523 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" podStartSLOduration=2.23843997 podStartE2EDuration="2.420503801s" totalImagesPullingTime="182.063831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:35:30.420302691 +0000 UTC m=+838.891777980" watchObservedRunningTime="2026-07-15 19:35:30.420503801 +0000 UTC m=+838.891979092" Jul 15 19:35:38.538069 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:38.538014 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:35:39.124278 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:39.124246 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:39.124562 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:39.124522 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:39.137096 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:39.137077 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:39.456392 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:39.456323 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz" Jul 15 19:35:56.368349 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:35:56.368315 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:36:27.048228 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:27.048142 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:36:27.149420 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:27.149382 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:36:48.697457 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:48.697424 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:36:48.797096 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:48.797066 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:36:49.921151 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:49.921116 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:36:49.982705 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:49.982666 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bcdb599d8-qjsks"] Jul 15 19:36:49.982956 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:49.982907 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-bcdb599d8-qjsks" podUID="06c5580d-f69d-4573-8eca-a30f63b0c511" containerName="authorino" containerID="cri-o://245b59f06f81b026cff2c5bd23df7261f66d08a618c2b246f741c5140202b854" gracePeriod=30 Jul 15 19:36:50.010058 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:50.010021 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:36:50.238806 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:50.238779 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bcdb599d8-qjsks" Jul 15 19:36:50.341825 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:50.341786 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgm48\" (UniqueName: \"kubernetes.io/projected/06c5580d-f69d-4573-8eca-a30f63b0c511-kube-api-access-tgm48\") pod \"06c5580d-f69d-4573-8eca-a30f63b0c511\" (UID: \"06c5580d-f69d-4573-8eca-a30f63b0c511\") " Jul 15 19:36:50.342004 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:50.341921 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06c5580d-f69d-4573-8eca-a30f63b0c511-tls-cert\") pod \"06c5580d-f69d-4573-8eca-a30f63b0c511\" (UID: \"06c5580d-f69d-4573-8eca-a30f63b0c511\") " Jul 15 19:36:50.344085 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:50.344057 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c5580d-f69d-4573-8eca-a30f63b0c511-kube-api-access-tgm48" (OuterVolumeSpecName: "kube-api-access-tgm48") pod "06c5580d-f69d-4573-8eca-a30f63b0c511" (UID: "06c5580d-f69d-4573-8eca-a30f63b0c511"). InnerVolumeSpecName "kube-api-access-tgm48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:36:50.352057 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:50.352035 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c5580d-f69d-4573-8eca-a30f63b0c511-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "06c5580d-f69d-4573-8eca-a30f63b0c511" (UID: "06c5580d-f69d-4573-8eca-a30f63b0c511"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:36:50.442905 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:50.442829 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgm48\" (UniqueName: \"kubernetes.io/projected/06c5580d-f69d-4573-8eca-a30f63b0c511-kube-api-access-tgm48\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:36:50.442905 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:50.442859 2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06c5580d-f69d-4573-8eca-a30f63b0c511-tls-cert\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:36:50.796558 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:50.796522 2564 generic.go:356] "Generic (PLEG): container finished" podID="06c5580d-f69d-4573-8eca-a30f63b0c511" containerID="245b59f06f81b026cff2c5bd23df7261f66d08a618c2b246f741c5140202b854" exitCode=0 Jul 15 19:36:50.796727 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:50.796570 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bcdb599d8-qjsks" Jul 15 19:36:50.796727 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:50.796611 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bcdb599d8-qjsks" event={"ID":"06c5580d-f69d-4573-8eca-a30f63b0c511","Type":"ContainerDied","Data":"245b59f06f81b026cff2c5bd23df7261f66d08a618c2b246f741c5140202b854"} Jul 15 19:36:50.796727 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:50.796656 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bcdb599d8-qjsks" event={"ID":"06c5580d-f69d-4573-8eca-a30f63b0c511","Type":"ContainerDied","Data":"793daf6cd25ebf9fb81b2971b79de3bb4638d4343deca1e4edf0f8c92c69d306"} Jul 15 19:36:50.796727 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:50.796673 2564 scope.go:117] "RemoveContainer" containerID="245b59f06f81b026cff2c5bd23df7261f66d08a618c2b246f741c5140202b854" Jul 15 19:36:50.806142 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:50.806119 2564 scope.go:117] "RemoveContainer" containerID="245b59f06f81b026cff2c5bd23df7261f66d08a618c2b246f741c5140202b854" Jul 15 19:36:50.806395 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:36:50.806375 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245b59f06f81b026cff2c5bd23df7261f66d08a618c2b246f741c5140202b854\": container with ID starting with 245b59f06f81b026cff2c5bd23df7261f66d08a618c2b246f741c5140202b854 not found: ID does not exist" containerID="245b59f06f81b026cff2c5bd23df7261f66d08a618c2b246f741c5140202b854" Jul 15 19:36:50.806435 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:50.806404 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245b59f06f81b026cff2c5bd23df7261f66d08a618c2b246f741c5140202b854"} err="failed to get container status \"245b59f06f81b026cff2c5bd23df7261f66d08a618c2b246f741c5140202b854\": rpc error: code = NotFound desc = could not find container \"245b59f06f81b026cff2c5bd23df7261f66d08a618c2b246f741c5140202b854\": container with ID starting with 245b59f06f81b026cff2c5bd23df7261f66d08a618c2b246f741c5140202b854 not found: ID does not exist" Jul 15 19:36:50.816673 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:50.816651 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bcdb599d8-qjsks"] Jul 15 19:36:50.820508 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:50.820487 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-bcdb599d8-qjsks"] Jul 15 19:36:51.526650 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:51.526615 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:36:52.060091 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:36:52.060059 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06c5580d-f69d-4573-8eca-a30f63b0c511" path="/var/lib/kubelet/pods/06c5580d-f69d-4573-8eca-a30f63b0c511/volumes" Jul 15 19:37:27.291507 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:37:27.291443 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:09.501902 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:09.501869 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:09.598832 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:09.598804 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6548747c7b-t8sd9"] Jul 15 19:38:09.599034 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:09.599013 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6548747c7b-t8sd9" podUID="9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2" containerName="manager" containerID="cri-o://39c2a178e348d925cffb3f3b8e81aedec4d9bb2af06f258a69f21d9c97ea97d5" gracePeriod=30 Jul 15 19:38:09.854027 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:09.854003 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6548747c7b-t8sd9" Jul 15 19:38:09.923249 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:09.923220 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2-webhook-cert\") pod \"9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2\" (UID: \"9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2\") " Jul 15 19:38:09.923406 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:09.923262 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crfld\" (UniqueName: \"kubernetes.io/projected/9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2-kube-api-access-crfld\") pod \"9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2\" (UID: \"9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2\") " Jul 15 19:38:09.925544 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:09.925510 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2" (UID: "9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:38:09.925544 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:09.925520 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2-kube-api-access-crfld" (OuterVolumeSpecName: "kube-api-access-crfld") pod "9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2" (UID: "9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2"). InnerVolumeSpecName "kube-api-access-crfld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:38:10.023995 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:10.023910 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crfld\" (UniqueName: \"kubernetes.io/projected/9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2-kube-api-access-crfld\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:38:10.023995 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:10.023939 2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2-webhook-cert\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:38:10.172133 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:10.172102 2564 generic.go:356] "Generic (PLEG): container finished" podID="9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2" containerID="39c2a178e348d925cffb3f3b8e81aedec4d9bb2af06f258a69f21d9c97ea97d5" exitCode=0 Jul 15 19:38:10.172278 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:10.172149 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6548747c7b-t8sd9" Jul 15 19:38:10.172278 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:10.172164 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6548747c7b-t8sd9" event={"ID":"9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2","Type":"ContainerDied","Data":"39c2a178e348d925cffb3f3b8e81aedec4d9bb2af06f258a69f21d9c97ea97d5"} Jul 15 19:38:10.172278 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:10.172205 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6548747c7b-t8sd9" event={"ID":"9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2","Type":"ContainerDied","Data":"4ca686ce8344b76ffbe2c519c456c64469c76e24c8e87789b4115fdff06e289f"} Jul 15 19:38:10.172278 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:10.172229 2564 scope.go:117] "RemoveContainer" containerID="39c2a178e348d925cffb3f3b8e81aedec4d9bb2af06f258a69f21d9c97ea97d5" Jul 15 19:38:10.181186 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:10.181167 2564 scope.go:117] "RemoveContainer" containerID="39c2a178e348d925cffb3f3b8e81aedec4d9bb2af06f258a69f21d9c97ea97d5" Jul 15 19:38:10.181485 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:38:10.181453 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c2a178e348d925cffb3f3b8e81aedec4d9bb2af06f258a69f21d9c97ea97d5\": container with ID starting with 39c2a178e348d925cffb3f3b8e81aedec4d9bb2af06f258a69f21d9c97ea97d5 not found: ID does not exist" containerID="39c2a178e348d925cffb3f3b8e81aedec4d9bb2af06f258a69f21d9c97ea97d5" Jul 15 19:38:10.181561 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:10.181507 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c2a178e348d925cffb3f3b8e81aedec4d9bb2af06f258a69f21d9c97ea97d5"} err="failed to get container status \"39c2a178e348d925cffb3f3b8e81aedec4d9bb2af06f258a69f21d9c97ea97d5\": rpc error: code = NotFound desc = could not find container \"39c2a178e348d925cffb3f3b8e81aedec4d9bb2af06f258a69f21d9c97ea97d5\": container with ID starting with 39c2a178e348d925cffb3f3b8e81aedec4d9bb2af06f258a69f21d9c97ea97d5 not found: ID does not exist" Jul 15 19:38:10.190644 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:10.190618 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6548747c7b-t8sd9"] Jul 15 19:38:10.202380 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:10.202354 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6548747c7b-t8sd9"] Jul 15 19:38:10.899953 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:10.899920 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:11.052882 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.052850 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6548747c7b-wwlbq"] Jul 15 19:38:11.053236 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.053222 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06c5580d-f69d-4573-8eca-a30f63b0c511" containerName="authorino" Jul 15 19:38:11.053290 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.053241 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="06c5580d-f69d-4573-8eca-a30f63b0c511" containerName="authorino" Jul 15 19:38:11.053290 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.053264 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2" containerName="manager" Jul 15 19:38:11.053290 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.053270 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2" containerName="manager" Jul 15 19:38:11.053387 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.053329 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2" containerName="manager" Jul 15 19:38:11.053387 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.053340 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="06c5580d-f69d-4573-8eca-a30f63b0c511" containerName="authorino" Jul 15 19:38:11.057576 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.057558 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6548747c7b-wwlbq" Jul 15 19:38:11.062964 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.062940 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 15 19:38:11.062964 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.062953 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8c6qr\"" Jul 15 19:38:11.063130 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.062970 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 15 19:38:11.074120 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.074098 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6548747c7b-wwlbq"] Jul 15 19:38:11.134850 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.134814 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6q9k\" (UniqueName: \"kubernetes.io/projected/f40bb241-197a-44f8-a070-053bd8158987-kube-api-access-j6q9k\") pod \"maas-controller-6548747c7b-wwlbq\" (UID: \"f40bb241-197a-44f8-a070-053bd8158987\") " pod="opendatahub/maas-controller-6548747c7b-wwlbq" Jul 15 19:38:11.135044 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.134887 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f40bb241-197a-44f8-a070-053bd8158987-webhook-cert\") pod \"maas-controller-6548747c7b-wwlbq\" (UID: \"f40bb241-197a-44f8-a070-053bd8158987\") " pod="opendatahub/maas-controller-6548747c7b-wwlbq" Jul 15 19:38:11.235753 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.235718 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6q9k\" (UniqueName: \"kubernetes.io/projected/f40bb241-197a-44f8-a070-053bd8158987-kube-api-access-j6q9k\") pod \"maas-controller-6548747c7b-wwlbq\" (UID: \"f40bb241-197a-44f8-a070-053bd8158987\") " pod="opendatahub/maas-controller-6548747c7b-wwlbq" Jul 15 19:38:11.235932 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.235770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f40bb241-197a-44f8-a070-053bd8158987-webhook-cert\") pod \"maas-controller-6548747c7b-wwlbq\" (UID: \"f40bb241-197a-44f8-a070-053bd8158987\") " pod="opendatahub/maas-controller-6548747c7b-wwlbq" Jul 15 19:38:11.238480 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.238454 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f40bb241-197a-44f8-a070-053bd8158987-webhook-cert\") pod \"maas-controller-6548747c7b-wwlbq\" (UID: \"f40bb241-197a-44f8-a070-053bd8158987\") " pod="opendatahub/maas-controller-6548747c7b-wwlbq" Jul 15 19:38:11.248794 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.248768 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6q9k\" (UniqueName: \"kubernetes.io/projected/f40bb241-197a-44f8-a070-053bd8158987-kube-api-access-j6q9k\") pod \"maas-controller-6548747c7b-wwlbq\" (UID: \"f40bb241-197a-44f8-a070-053bd8158987\") " pod="opendatahub/maas-controller-6548747c7b-wwlbq" Jul 15 19:38:11.368104 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.368068 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6548747c7b-wwlbq" Jul 15 19:38:11.517106 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.517017 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6548747c7b-wwlbq"] Jul 15 19:38:11.521532 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:38:11.521502 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40bb241_197a_44f8_a070_053bd8158987.slice/crio-34685c8211a720354691c7a11cadce827f409814f89d6e5c41e9abdfb84eb207 WatchSource:0}: Error finding container 34685c8211a720354691c7a11cadce827f409814f89d6e5c41e9abdfb84eb207: Status 404 returned error can't find the container with id 34685c8211a720354691c7a11cadce827f409814f89d6e5c41e9abdfb84eb207 Jul 15 19:38:11.523001 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:11.522976 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:38:12.062969 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:12.062927 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2" path="/var/lib/kubelet/pods/9b89e3c2-7ee4-46ed-8b51-6812a8c3bce2/volumes" Jul 15 19:38:12.183525 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:12.183464 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6548747c7b-wwlbq" event={"ID":"f40bb241-197a-44f8-a070-053bd8158987","Type":"ContainerStarted","Data":"67e0d23d16b2159932781e9e72302609a908f8e86de25a3f2b84967be5dc289c"} Jul 15 19:38:12.183719 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:12.183562 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6548747c7b-wwlbq" Jul 15 19:38:12.183719 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:12.183579 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6548747c7b-wwlbq" event={"ID":"f40bb241-197a-44f8-a070-053bd8158987","Type":"ContainerStarted","Data":"34685c8211a720354691c7a11cadce827f409814f89d6e5c41e9abdfb84eb207"} Jul 15 19:38:12.204780 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:12.204740 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6548747c7b-wwlbq" podStartSLOduration=0.749940693 podStartE2EDuration="1.204728207s" totalImagesPullingTime="454.787514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:12.203759591 +0000 UTC m=+1000.675234881" watchObservedRunningTime="2026-07-15 19:38:12.204728207 +0000 UTC m=+1000.676203496" Jul 15 19:38:23.190595 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:23.190562 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6548747c7b-wwlbq" Jul 15 19:38:54.110033 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:54.109998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:54.200748 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:54.200703 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:54.370513 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:38:54.370418 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:01.190272 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:39:01.190232 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:01.292201 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:39:01.292172 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:01.387667 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:39:01.387628 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:01.585811 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:39:01.585767 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:07.671094 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:39:07.671062 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:07.770575 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:39:07.770526 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:36.293849 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:39:36.293762 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:36.345766 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:39:36.345736 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:26.267419 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:41:26.267384 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:03.574228 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:03.574141 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps"] Jul 15 19:43:03.574838 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:03.574433 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" podUID="f8c07e9e-5cfd-4ab6-b28b-a015c53363c8" containerName="manager" containerID="cri-o://7dfc4b6e60d4e110ac669d0b74bb0cec3bb3b94ab0a39ba4bddef07646b081cc" gracePeriod=10 Jul 15 19:43:07.533645 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:07.533621 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" Jul 15 19:43:07.577579 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:07.577506 2564 generic.go:356] "Generic (PLEG): container finished" podID="f8c07e9e-5cfd-4ab6-b28b-a015c53363c8" containerID="7dfc4b6e60d4e110ac669d0b74bb0cec3bb3b94ab0a39ba4bddef07646b081cc" exitCode=0 Jul 15 19:43:07.577579 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:07.577539 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" event={"ID":"f8c07e9e-5cfd-4ab6-b28b-a015c53363c8","Type":"ContainerDied","Data":"7dfc4b6e60d4e110ac669d0b74bb0cec3bb3b94ab0a39ba4bddef07646b081cc"} Jul 15 19:43:07.577739 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:07.577601 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" event={"ID":"f8c07e9e-5cfd-4ab6-b28b-a015c53363c8","Type":"ContainerDied","Data":"abbf71801bf314475d8e5aa1552dcc29aaa56f4c20a8de6dec686ae8547533f6"} Jul 15 19:43:07.577739 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:07.577554 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps" Jul 15 19:43:07.577739 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:07.577624 2564 scope.go:117] "RemoveContainer" containerID="7dfc4b6e60d4e110ac669d0b74bb0cec3bb3b94ab0a39ba4bddef07646b081cc" Jul 15 19:43:07.581520 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:07.581498 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpglq\" (UniqueName: \"kubernetes.io/projected/f8c07e9e-5cfd-4ab6-b28b-a015c53363c8-kube-api-access-gpglq\") pod \"f8c07e9e-5cfd-4ab6-b28b-a015c53363c8\" (UID: \"f8c07e9e-5cfd-4ab6-b28b-a015c53363c8\") " Jul 15 19:43:07.581643 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:07.581573 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8c07e9e-5cfd-4ab6-b28b-a015c53363c8-extensions-socket-volume\") pod \"f8c07e9e-5cfd-4ab6-b28b-a015c53363c8\" (UID: \"f8c07e9e-5cfd-4ab6-b28b-a015c53363c8\") " Jul 15 19:43:07.582148 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:07.582066 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c07e9e-5cfd-4ab6-b28b-a015c53363c8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f8c07e9e-5cfd-4ab6-b28b-a015c53363c8" (UID: "f8c07e9e-5cfd-4ab6-b28b-a015c53363c8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:43:07.583727 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:07.583699 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c07e9e-5cfd-4ab6-b28b-a015c53363c8-kube-api-access-gpglq" (OuterVolumeSpecName: "kube-api-access-gpglq") pod "f8c07e9e-5cfd-4ab6-b28b-a015c53363c8" (UID: "f8c07e9e-5cfd-4ab6-b28b-a015c53363c8"). InnerVolumeSpecName "kube-api-access-gpglq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:43:07.587714 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:07.587696 2564 scope.go:117] "RemoveContainer" containerID="7dfc4b6e60d4e110ac669d0b74bb0cec3bb3b94ab0a39ba4bddef07646b081cc" Jul 15 19:43:07.587991 ip-10-0-141-225 kubenswrapper[2564]: E0715 19:43:07.587971 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dfc4b6e60d4e110ac669d0b74bb0cec3bb3b94ab0a39ba4bddef07646b081cc\": container with ID starting with 7dfc4b6e60d4e110ac669d0b74bb0cec3bb3b94ab0a39ba4bddef07646b081cc not found: ID does not exist" containerID="7dfc4b6e60d4e110ac669d0b74bb0cec3bb3b94ab0a39ba4bddef07646b081cc" Jul 15 19:43:07.588053 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:07.588003 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dfc4b6e60d4e110ac669d0b74bb0cec3bb3b94ab0a39ba4bddef07646b081cc"} err="failed to get container status \"7dfc4b6e60d4e110ac669d0b74bb0cec3bb3b94ab0a39ba4bddef07646b081cc\": rpc error: code = NotFound desc = could not find container \"7dfc4b6e60d4e110ac669d0b74bb0cec3bb3b94ab0a39ba4bddef07646b081cc\": container with ID starting with 7dfc4b6e60d4e110ac669d0b74bb0cec3bb3b94ab0a39ba4bddef07646b081cc not found: ID does not exist" Jul 15 19:43:07.683031 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:07.683005 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpglq\" (UniqueName: \"kubernetes.io/projected/f8c07e9e-5cfd-4ab6-b28b-a015c53363c8-kube-api-access-gpglq\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:43:07.683031 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:07.683030 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8c07e9e-5cfd-4ab6-b28b-a015c53363c8-extensions-socket-volume\") on node \"ip-10-0-141-225.ec2.internal\" DevicePath \"\"" Jul 15 19:43:07.899537 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:07.899510 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps"] Jul 15 19:43:07.905976 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:07.905953 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sx5ps"] Jul 15 19:43:08.060202 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:43:08.060164 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8c07e9e-5cfd-4ab6-b28b-a015c53363c8" path="/var/lib/kubelet/pods/f8c07e9e-5cfd-4ab6-b28b-a015c53363c8/volumes" Jul 15 19:44:04.819118 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:04.819032 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2lgr"] Jul 15 19:44:04.819709 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:04.819423 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c07e9e-5cfd-4ab6-b28b-a015c53363c8" containerName="manager" Jul 15 19:44:04.819709 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:04.819433 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c07e9e-5cfd-4ab6-b28b-a015c53363c8" containerName="manager" Jul 15 19:44:04.819709 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:04.819517 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c07e9e-5cfd-4ab6-b28b-a015c53363c8" containerName="manager" Jul 15 19:44:04.822546 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:04.822529 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2lgr" Jul 15 19:44:04.832834 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:04.832810 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m5c79\"" Jul 15 19:44:04.864484 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:04.864443 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2lgr"] Jul 15 19:44:04.873835 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:04.873805 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqdgg\" (UniqueName: \"kubernetes.io/projected/e0dad896-b9f3-4d2a-b5e1-b92dda4828e1-kube-api-access-zqdgg\") pod \"kuadrant-operator-controller-manager-86878c769f-d2lgr\" (UID: \"e0dad896-b9f3-4d2a-b5e1-b92dda4828e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2lgr" Jul 15 19:44:04.873948 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:04.873933 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0dad896-b9f3-4d2a-b5e1-b92dda4828e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-d2lgr\" (UID: \"e0dad896-b9f3-4d2a-b5e1-b92dda4828e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2lgr" Jul 15 19:44:04.974993 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:04.974957 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0dad896-b9f3-4d2a-b5e1-b92dda4828e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-d2lgr\" (UID: \"e0dad896-b9f3-4d2a-b5e1-b92dda4828e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2lgr" Jul 15 19:44:04.975159 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:04.975027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqdgg\" (UniqueName: \"kubernetes.io/projected/e0dad896-b9f3-4d2a-b5e1-b92dda4828e1-kube-api-access-zqdgg\") pod \"kuadrant-operator-controller-manager-86878c769f-d2lgr\" (UID: \"e0dad896-b9f3-4d2a-b5e1-b92dda4828e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2lgr" Jul 15 19:44:04.975339 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:04.975318 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0dad896-b9f3-4d2a-b5e1-b92dda4828e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-d2lgr\" (UID: \"e0dad896-b9f3-4d2a-b5e1-b92dda4828e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2lgr" Jul 15 19:44:04.988441 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:04.988407 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqdgg\" (UniqueName: \"kubernetes.io/projected/e0dad896-b9f3-4d2a-b5e1-b92dda4828e1-kube-api-access-zqdgg\") pod \"kuadrant-operator-controller-manager-86878c769f-d2lgr\" (UID: \"e0dad896-b9f3-4d2a-b5e1-b92dda4828e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2lgr" Jul 15 19:44:05.132372 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:05.132291 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2lgr" Jul 15 19:44:05.279298 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:05.279271 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2lgr"] Jul 15 19:44:05.282213 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:44:05.282186 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0dad896_b9f3_4d2a_b5e1_b92dda4828e1.slice/crio-d41627363d60e646fbc3d8e44bc5ebb40bb00dc39d37e42c8d046b56c2f72ce9 WatchSource:0}: Error finding container d41627363d60e646fbc3d8e44bc5ebb40bb00dc39d37e42c8d046b56c2f72ce9: Status 404 returned error can't find the container with id d41627363d60e646fbc3d8e44bc5ebb40bb00dc39d37e42c8d046b56c2f72ce9 Jul 15 19:44:05.839301 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:05.839264 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2lgr" event={"ID":"e0dad896-b9f3-4d2a-b5e1-b92dda4828e1","Type":"ContainerStarted","Data":"d2de7bc480b7087fe647b80d55aa00e94f56c9ac0fbb548acfa73fd83869f489"} Jul 15 19:44:05.839737 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:05.839316 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2lgr" event={"ID":"e0dad896-b9f3-4d2a-b5e1-b92dda4828e1","Type":"ContainerStarted","Data":"d41627363d60e646fbc3d8e44bc5ebb40bb00dc39d37e42c8d046b56c2f72ce9"} Jul 15 19:44:05.839737 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:05.839364 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2lgr" Jul 15 19:44:05.863599 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:05.863550 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2lgr" podStartSLOduration=1.863532787 podStartE2EDuration="1.863532787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:44:05.862550682 +0000 UTC m=+1354.334025985" watchObservedRunningTime="2026-07-15 19:44:05.863532787 +0000 UTC m=+1354.335008075" Jul 15 19:44:16.845433 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:44:16.845403 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2lgr" Jul 15 19:45:00.171204 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:45:00.171172 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735745-w85hb"] Jul 15 19:45:00.174771 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:45:00.174744 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735745-w85hb" Jul 15 19:45:00.177453 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:45:00.177434 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nsffg\"" Jul 15 19:45:00.184096 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:45:00.184077 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735745-w85hb" Jul 15 19:45:00.190422 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:45:00.190400 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735745-w85hb"] Jul 15 19:45:00.343938 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:45:00.343909 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735745-w85hb"] Jul 15 19:45:00.347847 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:45:00.347811 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9da8f2_999c_4db4_b241_0a1bc744be93.slice/crio-cc5ee97aa381664afc2908ac3c0fb5c16ec0b3d0c613ce1ee8b752dbb6b8dc99 WatchSource:0}: Error finding container cc5ee97aa381664afc2908ac3c0fb5c16ec0b3d0c613ce1ee8b752dbb6b8dc99: Status 404 returned error can't find the container with id cc5ee97aa381664afc2908ac3c0fb5c16ec0b3d0c613ce1ee8b752dbb6b8dc99 Jul 15 19:45:00.349933 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:45:00.349909 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:45:01.091779 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:45:01.091739 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735745-w85hb" event={"ID":"1f9da8f2-999c-4db4-b241-0a1bc744be93","Type":"ContainerStarted","Data":"cc5ee97aa381664afc2908ac3c0fb5c16ec0b3d0c613ce1ee8b752dbb6b8dc99"} Jul 15 19:45:03.102742 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:45:03.102706 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735745-w85hb" event={"ID":"1f9da8f2-999c-4db4-b241-0a1bc744be93","Type":"ContainerStarted","Data":"0f81c826151580f30f1cd7c57fe71c7d7c186ebbc57628411ad1a89b7fe40b5f"} Jul 15 19:45:03.124973 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:45:03.124930 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735745-w85hb" podStartSLOduration=1.10282669 podStartE2EDuration="3.124913927s" totalImagesPullingTime="2.022087237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:45:03.122109843 +0000 UTC m=+1411.593585123" watchObservedRunningTime="2026-07-15 19:45:03.124913927 +0000 UTC m=+1411.596389216" Jul 15 19:45:04.109220 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:45:04.109189 2564 generic.go:356] "Generic (PLEG): container finished" podID="1f9da8f2-999c-4db4-b241-0a1bc744be93" containerID="0f81c826151580f30f1cd7c57fe71c7d7c186ebbc57628411ad1a89b7fe40b5f" exitCode=0 Jul 15 19:45:04.109655 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:45:04.109233 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735745-w85hb" event={"ID":"1f9da8f2-999c-4db4-b241-0a1bc744be93","Type":"ContainerDied","Data":"0f81c826151580f30f1cd7c57fe71c7d7c186ebbc57628411ad1a89b7fe40b5f"} Jul 15 19:45:05.252161 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:45:05.252139 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735745-w85hb" Jul 15 19:45:06.121076 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:45:06.121046 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735745-w85hb" Jul 15 19:45:06.121260 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:45:06.121075 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735745-w85hb" event={"ID":"1f9da8f2-999c-4db4-b241-0a1bc744be93","Type":"ContainerDied","Data":"cc5ee97aa381664afc2908ac3c0fb5c16ec0b3d0c613ce1ee8b752dbb6b8dc99"} Jul 15 19:45:06.121260 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:45:06.121131 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc5ee97aa381664afc2908ac3c0fb5c16ec0b3d0c613ce1ee8b752dbb6b8dc99" Jul 15 19:46:06.981289 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:46:06.981254 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:07.055839 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:46:07.055810 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:15.129070 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:47:15.129037 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:23.018735 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:23.018698 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-nlmpm_dcb8c706-6b2d-4df0-99b8-294b0e3f4221/manager/0.log" Jul 15 19:50:23.151960 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:23.151927 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-76jrd_c1430a43-7fd3-4989-ab2d-1047ea045c53/manager/0.log" Jul 15 19:50:23.289858 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:23.289785 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6548747c7b-wwlbq_f40bb241-197a-44f8-a070-053bd8158987/manager/0.log" Jul 15 19:50:23.418143 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:23.418112 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vxlh2_d0413cf3-332b-4ea7-8ca2-a90586a494fc/server/0.log" Jul 15 19:50:23.546126 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:23.546051 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qtll6_594a348b-5c74-4c66-9f9b-f94cfacf9fb6/manager/0.log" Jul 15 19:50:23.958236 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:23.958138 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df875cfdf-jrzwl_70c52184-a5a3-4c81-9d01-35650b2fb7b0/copy-manifests/0.log" Jul 15 19:50:23.975018 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:23.974983 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df875cfdf-jrzwl_70c52184-a5a3-4c81-9d01-35650b2fb7b0/manager/0.log" Jul 15 19:50:24.380064 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:24.380037 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-86bc485959-qjbnl_c873091d-6ab8-4c72-85ab-f5bd0ddc5365/maas-api/0.log" Jul 15 19:50:24.510055 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:24.510028 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29735745-w85hb_1f9da8f2-999c-4db4-b241-0a1bc744be93/cleanup/0.log" Jul 15 19:50:24.648356 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:24.648279 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-5q7p7_86420d42-d2e5-4a51-87e2-52d2ef1a7a36/postgres/0.log" Jul 15 19:50:26.228858 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:26.228822 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-h52kx_c29b17f7-d92e-4aaa-a2a8-07aa4514f1a9/manager/0.log" Jul 15 19:50:26.590975 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:26.590869 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-t5wcq_1ab19b99-0b01-4170-beb7-a57491ee7378/registry-server/0.log" Jul 15 19:50:26.721380 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:26.721346 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-d2lgr_e0dad896-b9f3-4d2a-b5e1-b92dda4828e1/manager/0.log" Jul 15 19:50:27.331846 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:27.331816 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4llb5h_07f8ab2c-946b-4be7-9708-1a6f8041d68c/istio-proxy/0.log" Jul 15 19:50:27.701775 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:27.701693 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d4465d888-r29kn_3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3/kube-auth-proxy/0.log" Jul 15 19:50:27.833795 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:27.833765 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-srxrd_176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd/istio-proxy/1.log" Jul 15 19:50:27.963396 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:27.963364 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-7gkbp_a3701cc2-6375-4df3-99f4-f15804d6e560/payload-pre-processing/0.log" Jul 15 19:50:28.574399 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:28.574360 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-kxf5m_9fd73d95-2a73-43d2-ad0c-6d026ea94931/main/0.log" Jul 15 19:50:28.961086 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:28.961065 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd8djz_6a63f9a5-f0f3-4c36-bf79-7b1c94042ffb/main/0.log" Jul 15 19:50:35.391669 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.391632 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-crt2f/must-gather-8mvjh"] Jul 15 19:50:35.392171 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.392152 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f9da8f2-999c-4db4-b241-0a1bc744be93" containerName="cleanup" Jul 15 19:50:35.392252 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.392174 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f9da8f2-999c-4db4-b241-0a1bc744be93" containerName="cleanup" Jul 15 19:50:35.392320 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.392289 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f9da8f2-999c-4db4-b241-0a1bc744be93" containerName="cleanup" Jul 15 19:50:35.395384 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.395353 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crt2f/must-gather-8mvjh" Jul 15 19:50:35.397681 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.397661 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-crt2f\"/\"openshift-service-ca.crt\"" Jul 15 19:50:35.397769 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.397755 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-crt2f\"/\"kube-root-ca.crt\"" Jul 15 19:50:35.398126 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.398112 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-crt2f\"/\"default-dockercfg-vddw7\"" Jul 15 19:50:35.416857 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.416835 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-crt2f/must-gather-8mvjh"] Jul 15 19:50:35.423166 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.423142 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:35.448004 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.447979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a2434b18-f6fb-40a8-8e62-5249ee830f0a-must-gather-output\") pod \"must-gather-8mvjh\" (UID: \"a2434b18-f6fb-40a8-8e62-5249ee830f0a\") " pod="openshift-must-gather-crt2f/must-gather-8mvjh" Jul 15 19:50:35.448108 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.448088 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdbbz\" (UniqueName: \"kubernetes.io/projected/a2434b18-f6fb-40a8-8e62-5249ee830f0a-kube-api-access-bdbbz\") pod \"must-gather-8mvjh\" (UID: \"a2434b18-f6fb-40a8-8e62-5249ee830f0a\") " pod="openshift-must-gather-crt2f/must-gather-8mvjh" Jul 15 19:50:35.504919 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.504890 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:35.549344 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.549317 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdbbz\" (UniqueName: \"kubernetes.io/projected/a2434b18-f6fb-40a8-8e62-5249ee830f0a-kube-api-access-bdbbz\") pod \"must-gather-8mvjh\" (UID: \"a2434b18-f6fb-40a8-8e62-5249ee830f0a\") " pod="openshift-must-gather-crt2f/must-gather-8mvjh" Jul 15 19:50:35.549458 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.549359 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a2434b18-f6fb-40a8-8e62-5249ee830f0a-must-gather-output\") pod \"must-gather-8mvjh\" (UID: \"a2434b18-f6fb-40a8-8e62-5249ee830f0a\") " pod="openshift-must-gather-crt2f/must-gather-8mvjh" Jul 15 19:50:35.549685 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.549668 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a2434b18-f6fb-40a8-8e62-5249ee830f0a-must-gather-output\") pod \"must-gather-8mvjh\" (UID: \"a2434b18-f6fb-40a8-8e62-5249ee830f0a\") " pod="openshift-must-gather-crt2f/must-gather-8mvjh" Jul 15 19:50:35.559924 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.559894 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdbbz\" (UniqueName: \"kubernetes.io/projected/a2434b18-f6fb-40a8-8e62-5249ee830f0a-kube-api-access-bdbbz\") pod \"must-gather-8mvjh\" (UID: \"a2434b18-f6fb-40a8-8e62-5249ee830f0a\") " pod="openshift-must-gather-crt2f/must-gather-8mvjh" Jul 15 19:50:35.704823 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.704745 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crt2f/must-gather-8mvjh" Jul 15 19:50:35.839930 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.839873 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-crt2f/must-gather-8mvjh"] Jul 15 19:50:35.842240 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:50:35.842214 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2434b18_f6fb_40a8_8e62_5249ee830f0a.slice/crio-f1f1fc56fd7dfb8a174346593c8ae4565a5a4332fc551e80d44397e9e9821c73 WatchSource:0}: Error finding container f1f1fc56fd7dfb8a174346593c8ae4565a5a4332fc551e80d44397e9e9821c73: Status 404 returned error can't find the container with id f1f1fc56fd7dfb8a174346593c8ae4565a5a4332fc551e80d44397e9e9821c73 Jul 15 19:50:35.844141 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:35.844124 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:50:36.683929 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:36.683894 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crt2f/must-gather-8mvjh" event={"ID":"a2434b18-f6fb-40a8-8e62-5249ee830f0a","Type":"ContainerStarted","Data":"f1f1fc56fd7dfb8a174346593c8ae4565a5a4332fc551e80d44397e9e9821c73"} Jul 15 19:50:37.693305 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:37.693264 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crt2f/must-gather-8mvjh" event={"ID":"a2434b18-f6fb-40a8-8e62-5249ee830f0a","Type":"ContainerStarted","Data":"bfb53e26259c0d6c4adfab0fdb56166ed1feb65d46f886d62be446ba9946bfaf"} Jul 15 19:50:37.693771 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:37.693320 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crt2f/must-gather-8mvjh" event={"ID":"a2434b18-f6fb-40a8-8e62-5249ee830f0a","Type":"ContainerStarted","Data":"ab9e150779e34bea524146475fe0208233643beb1aaca1834dacc17f7e5f3e5d"} Jul 15 19:50:37.711884 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:37.711835 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-crt2f/must-gather-8mvjh" podStartSLOduration=1.8397188230000001 podStartE2EDuration="2.711820322s" totalImagesPullingTime="872.101499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:50:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:50:37.710281683 +0000 UTC m=+1746.181756969" watchObservedRunningTime="2026-07-15 19:50:37.711820322 +0000 UTC m=+1746.183295611" Jul 15 19:50:38.457122 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:38.457094 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pz2h5_e089ab9b-564c-4a0d-9b14-4bce14a4f070/global-pull-secret-syncer/0.log" Jul 15 19:50:38.553596 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:38.553566 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g6pfk_d2a3fc5a-d407-4ad9-9862-e5fcda33d18e/konnectivity-agent/0.log" Jul 15 19:50:38.689201 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:38.689162 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-225.ec2.internal_90a23a491012cfceb92edd519600eade/haproxy/0.log" Jul 15 19:50:44.490594 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:44.490559 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-h52kx_c29b17f7-d92e-4aaa-a2a8-07aa4514f1a9/manager/0.log" Jul 15 19:50:44.601434 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:44.601378 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-t5wcq_1ab19b99-0b01-4170-beb7-a57491ee7378/registry-server/0.log" Jul 15 19:50:44.671785 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:44.671750 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-d2lgr_e0dad896-b9f3-4d2a-b5e1-b92dda4828e1/manager/0.log" Jul 15 19:50:46.324249 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:46.324212 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_52bcde0a-20d0-49fc-b970-8d707cf2c5dd/alertmanager/0.log" Jul 15 19:50:46.351116 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:46.351081 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_52bcde0a-20d0-49fc-b970-8d707cf2c5dd/config-reloader/0.log" Jul 15 19:50:46.377642 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:46.377614 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_52bcde0a-20d0-49fc-b970-8d707cf2c5dd/kube-rbac-proxy-web/0.log" Jul 15 19:50:46.404900 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:46.404863 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_52bcde0a-20d0-49fc-b970-8d707cf2c5dd/kube-rbac-proxy/0.log" Jul 15 19:50:46.431836 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:46.431790 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_52bcde0a-20d0-49fc-b970-8d707cf2c5dd/kube-rbac-proxy-metric/0.log" Jul 15 19:50:46.458584 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:46.458554 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_52bcde0a-20d0-49fc-b970-8d707cf2c5dd/prom-label-proxy/0.log" Jul 15 19:50:46.486117 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:46.486075 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_52bcde0a-20d0-49fc-b970-8d707cf2c5dd/init-config-reloader/0.log" Jul 15 19:50:46.780229 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:46.780197 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6gpmh_681ecdbc-c87a-4a18-b17d-af54d2268c06/node-exporter/0.log" Jul 15 19:50:46.806824 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:46.806799 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6gpmh_681ecdbc-c87a-4a18-b17d-af54d2268c06/kube-rbac-proxy/0.log" Jul 15 19:50:46.829295 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:46.829265 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6gpmh_681ecdbc-c87a-4a18-b17d-af54d2268c06/init-textfile/0.log" Jul 15 19:50:47.943656 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:47.943617 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg"] Jul 15 19:50:47.950493 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:47.950436 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:47.971969 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:47.971935 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg"] Jul 15 19:50:48.086614 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.086585 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d805d4af-eac0-4856-b56a-d671e6a95b41-sys\") pod \"perf-node-gather-daemonset-2trtg\" (UID: \"d805d4af-eac0-4856-b56a-d671e6a95b41\") " pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:48.086809 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.086638 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d805d4af-eac0-4856-b56a-d671e6a95b41-proc\") pod \"perf-node-gather-daemonset-2trtg\" (UID: \"d805d4af-eac0-4856-b56a-d671e6a95b41\") " pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:48.086809 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.086725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d805d4af-eac0-4856-b56a-d671e6a95b41-podres\") pod \"perf-node-gather-daemonset-2trtg\" (UID: \"d805d4af-eac0-4856-b56a-d671e6a95b41\") " pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:48.086809 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.086761 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d805d4af-eac0-4856-b56a-d671e6a95b41-lib-modules\") pod \"perf-node-gather-daemonset-2trtg\" (UID: \"d805d4af-eac0-4856-b56a-d671e6a95b41\") " pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:48.086809 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.086795 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv7f6\" (UniqueName: \"kubernetes.io/projected/d805d4af-eac0-4856-b56a-d671e6a95b41-kube-api-access-vv7f6\") pod \"perf-node-gather-daemonset-2trtg\" (UID: \"d805d4af-eac0-4856-b56a-d671e6a95b41\") " pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:48.187438 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.187400 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv7f6\" (UniqueName: \"kubernetes.io/projected/d805d4af-eac0-4856-b56a-d671e6a95b41-kube-api-access-vv7f6\") pod \"perf-node-gather-daemonset-2trtg\" (UID: \"d805d4af-eac0-4856-b56a-d671e6a95b41\") " pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:48.187656 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.187496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d805d4af-eac0-4856-b56a-d671e6a95b41-sys\") pod \"perf-node-gather-daemonset-2trtg\" (UID: \"d805d4af-eac0-4856-b56a-d671e6a95b41\") " pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:48.187656 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.187566 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d805d4af-eac0-4856-b56a-d671e6a95b41-proc\") pod \"perf-node-gather-daemonset-2trtg\" (UID: \"d805d4af-eac0-4856-b56a-d671e6a95b41\") " pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:48.187656 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.187648 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d805d4af-eac0-4856-b56a-d671e6a95b41-podres\") pod \"perf-node-gather-daemonset-2trtg\" (UID: \"d805d4af-eac0-4856-b56a-d671e6a95b41\") " pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:48.187860 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.187664 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d805d4af-eac0-4856-b56a-d671e6a95b41-proc\") pod \"perf-node-gather-daemonset-2trtg\" (UID: \"d805d4af-eac0-4856-b56a-d671e6a95b41\") " pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:48.187860 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.187726 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d805d4af-eac0-4856-b56a-d671e6a95b41-lib-modules\") pod \"perf-node-gather-daemonset-2trtg\" (UID: \"d805d4af-eac0-4856-b56a-d671e6a95b41\") " pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:48.187981 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.187887 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d805d4af-eac0-4856-b56a-d671e6a95b41-lib-modules\") pod \"perf-node-gather-daemonset-2trtg\" (UID: \"d805d4af-eac0-4856-b56a-d671e6a95b41\") " pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:48.187981 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.187950 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d805d4af-eac0-4856-b56a-d671e6a95b41-podres\") pod \"perf-node-gather-daemonset-2trtg\" (UID: \"d805d4af-eac0-4856-b56a-d671e6a95b41\") " pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:48.188102 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.188049 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d805d4af-eac0-4856-b56a-d671e6a95b41-sys\") pod \"perf-node-gather-daemonset-2trtg\" (UID: \"d805d4af-eac0-4856-b56a-d671e6a95b41\") " pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:48.197678 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.197612 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv7f6\" (UniqueName: \"kubernetes.io/projected/d805d4af-eac0-4856-b56a-d671e6a95b41-kube-api-access-vv7f6\") pod \"perf-node-gather-daemonset-2trtg\" (UID: \"d805d4af-eac0-4856-b56a-d671e6a95b41\") " pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:48.264666 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.264628 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:48.448090 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.447996 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg"] Jul 15 19:50:48.453334 ip-10-0-141-225 kubenswrapper[2564]: W0715 19:50:48.453305 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd805d4af_eac0_4856_b56a_d671e6a95b41.slice/crio-e9d6c0e9a3db082c21caad314d223b45cc767fa19d8947b13536d56e728e415f WatchSource:0}: Error finding container e9d6c0e9a3db082c21caad314d223b45cc767fa19d8947b13536d56e728e415f: Status 404 returned error can't find the container with id e9d6c0e9a3db082c21caad314d223b45cc767fa19d8947b13536d56e728e415f Jul 15 19:50:48.773402 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.773363 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" event={"ID":"d805d4af-eac0-4856-b56a-d671e6a95b41","Type":"ContainerStarted","Data":"ed6f1c2657fc74586e749ea24a798e46aab39b52bc1b4edbc63fe92df5ee4ee7"} Jul 15 19:50:48.774141 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.774104 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" event={"ID":"d805d4af-eac0-4856-b56a-d671e6a95b41","Type":"ContainerStarted","Data":"e9d6c0e9a3db082c21caad314d223b45cc767fa19d8947b13536d56e728e415f"} Jul 15 19:50:48.774267 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.774150 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:48.794144 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:48.794078 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" podStartSLOduration=1.794059588 podStartE2EDuration="1.794059588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:50:48.791317441 +0000 UTC m=+1757.262792741" watchObservedRunningTime="2026-07-15 19:50:48.794059588 +0000 UTC m=+1757.265534878" Jul 15 19:50:52.156340 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:52.156310 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xtdvl_6c110945-616b-4146-ac13-c20e4c84ef1e/dns/0.log" Jul 15 19:50:52.187425 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:52.187383 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xtdvl_6c110945-616b-4146-ac13-c20e4c84ef1e/kube-rbac-proxy/0.log" Jul 15 19:50:52.262947 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:52.262919 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lrfwx_a323a5a4-1f3f-49b5-ae58-8b5c90592e5a/dns-node-resolver/0.log" Jul 15 19:50:53.030884 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:53.030855 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n6mjl_98333cd6-ba91-4b38-bf05-64e47d2f0251/node-ca/0.log" Jul 15 19:50:54.152272 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:54.152241 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4llb5h_07f8ab2c-946b-4be7-9708-1a6f8041d68c/istio-proxy/0.log" Jul 15 19:50:54.271856 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:54.271809 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d4465d888-r29kn_3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3/kube-auth-proxy/0.log" Jul 15 19:50:54.307482 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:54.307440 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-srxrd_176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd/istio-proxy/0.log" Jul 15 19:50:54.338584 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:54.338552 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-srxrd_176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd/istio-proxy/1.log" Jul 15 19:50:54.371069 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:54.371042 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-7gkbp_a3701cc2-6375-4df3-99f4-f15804d6e560/payload-pre-processing/0.log" Jul 15 19:50:54.792581 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:54.792543 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-crt2f/perf-node-gather-daemonset-2trtg" Jul 15 19:50:55.097681 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:55.097603 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p5kfg_26ff859f-ec36-43d8-888d-bba78cf3dffb/serve-healthcheck-canary/0.log" Jul 15 19:50:55.702464 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:55.702437 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2gvbg_d91f01df-53be-4664-98ac-3a960fc0aebd/kube-rbac-proxy/0.log" Jul 15 19:50:55.725408 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:55.725381 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2gvbg_d91f01df-53be-4664-98ac-3a960fc0aebd/exporter/0.log" Jul 15 19:50:55.750425 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:55.750399 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2gvbg_d91f01df-53be-4664-98ac-3a960fc0aebd/extractor/0.log" Jul 15 19:50:58.632975 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:58.632902 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4llb5h_07f8ab2c-946b-4be7-9708-1a6f8041d68c/istio-proxy/0.log" Jul 15 19:50:58.695012 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:58.694986 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-nlmpm_dcb8c706-6b2d-4df0-99b8-294b0e3f4221/manager/0.log" Jul 15 19:50:58.717575 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:58.717548 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d4465d888-r29kn_3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3/kube-auth-proxy/0.log" Jul 15 19:50:58.743786 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:58.743759 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-srxrd_176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd/istio-proxy/0.log" Jul 15 19:50:58.791573 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:58.791535 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-srxrd_176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd/istio-proxy/1.log" Jul 15 19:50:58.793406 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:58.793385 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-76jrd_c1430a43-7fd3-4989-ab2d-1047ea045c53/manager/0.log" Jul 15 19:50:58.819338 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:58.819311 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-7gkbp_a3701cc2-6375-4df3-99f4-f15804d6e560/payload-pre-processing/0.log" Jul 15 19:50:58.867311 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:58.867282 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6548747c7b-wwlbq_f40bb241-197a-44f8-a070-053bd8158987/manager/0.log" Jul 15 19:50:58.905092 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:58.905017 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vxlh2_d0413cf3-332b-4ea7-8ca2-a90586a494fc/server/0.log" Jul 15 19:50:58.996229 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:58.996199 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qtll6_594a348b-5c74-4c66-9f9b-f94cfacf9fb6/manager/0.log" Jul 15 19:50:59.232644 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:59.232621 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df875cfdf-jrzwl_70c52184-a5a3-4c81-9d01-35650b2fb7b0/manager/0.log" Jul 15 19:50:59.271637 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:50:59.271613 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df875cfdf-jrzwl_70c52184-a5a3-4c81-9d01-35650b2fb7b0/copy-manifests/0.log" Jul 15 19:51:00.087689 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:51:00.087656 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4llb5h_07f8ab2c-946b-4be7-9708-1a6f8041d68c/istio-proxy/0.log" Jul 15 19:51:00.176899 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:51:00.176872 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d4465d888-r29kn_3fd1b2ec-4ec4-4026-b4c7-c3dab455b9e3/kube-auth-proxy/0.log" Jul 15 19:51:00.199450 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:51:00.199418 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-srxrd_176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd/istio-proxy/0.log" Jul 15 19:51:00.228378 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:51:00.228343 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-srxrd_176fee8a-e6dd-4e5b-8e43-12fe83fe9ffd/istio-proxy/1.log" Jul 15 19:51:00.247586 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:51:00.247554 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-7gkbp_a3701cc2-6375-4df3-99f4-f15804d6e560/payload-pre-processing/0.log" Jul 15 19:51:00.645013 ip-10-0-141-225 kubenswrapper[2564]: I0715 19:51:00.644985 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-cxqn5_fd709f77-fa25-477c-b555-b3659f0d5555/openshift-lws-operator/0.log"