Jul 16 12:06:59.606608 ip-10-0-143-101 systemd[1]: Starting Kubernetes Kubelet... Jul 16 12:07:00.129912 ip-10-0-143-101 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 12:07:00.129912 ip-10-0-143-101 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 12:07:00.129912 ip-10-0-143-101 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 12:07:00.129912 ip-10-0-143-101 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 12:07:00.129912 ip-10-0-143-101 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 12:07:00.129912 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.078430 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 12:07:00.129912 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086104 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:07:00.129912 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086121 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:07:00.129912 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086125 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:07:00.129912 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086128 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:07:00.129912 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086131 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:07:00.129912 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086134 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:07:00.129912 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086137 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:07:00.129912 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086139 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086142 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086145 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086147 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086150 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086153 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086156 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086158 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086161 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086164 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086174 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086177 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086181 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086184 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086187 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086189 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086192 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086195 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086197 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086200 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:07:00.397902 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086204 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:07:00.170576 ip-10-0-143-101 systemd[1]: Started Kubernetes Kubelet. Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086206 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086209 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086212 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086214 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086217 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086220 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086222 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086225 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086227 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086230 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086234 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086236 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086239 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086242 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086245 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086248 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086250 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086253 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086256 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:07:00.409720 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086258 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086261 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086263 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086266 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086271 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086275 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086278 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086281 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086284 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086288 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086291 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086294 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086297 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086300 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086303 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086305 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086308 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086311 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086315 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:07:00.410649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086318 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086333 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086336 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086339 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086341 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086344 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086347 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086350 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086352 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086355 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086357 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086361 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086365 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086368 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086371 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086373 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086377 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086381 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086385 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:07:00.411649 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086389 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086391 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086394 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086396 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086399 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086402 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086404 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086407 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086409 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086412 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086414 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086417 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086424 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.086426 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.090660 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.090678 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.091645 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.091651 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.091655 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.091663 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:07:00.412265 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.091669 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.091673 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.091677 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.091682 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.091686 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.091691 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.091695 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.091699 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.091703 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.091708 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.091712 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.091719 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.091727 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092718 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092726 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092734 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092738 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092750 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092755 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:07:00.413087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092759 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092763 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092767 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092771 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092775 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092779 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092783 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092787 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092791 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092796 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092812 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092820 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092823 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092826 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092829 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092831 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092835 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092837 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092840 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092842 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:07:00.413706 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092845 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092848 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092851 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092853 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092856 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092861 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092864 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092866 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092869 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092872 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092875 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092878 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092881 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092884 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092887 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092890 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092892 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092895 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092898 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092900 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:07:00.414566 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092903 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092906 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092908 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092911 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092913 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092918 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092923 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092927 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092929 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092932 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092934 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092937 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092939 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092942 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092945 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092947 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092950 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092952 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092956 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092958 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:07:00.415194 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092961 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092963 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092966 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092971 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092974 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092977 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092980 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092983 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092987 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092991 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092993 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092996 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.092998 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093001 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093412 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093418 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093421 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093424 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093427 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:07:00.415827 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093430 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093433 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093435 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093438 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093441 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093444 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093446 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093450 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093454 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093456 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093461 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093465 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093468 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093471 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093474 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093476 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093485 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093489 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093491 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:07:00.416410 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093494 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093497 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093499 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093502 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093504 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093507 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093509 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093512 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093515 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093518 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093521 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093523 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093526 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093528 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093531 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093533 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093536 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093538 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093541 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093543 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:07:00.417011 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093546 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093548 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093550 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093553 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093555 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093558 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093561 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093563 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093566 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093571 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093573 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093576 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093578 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093581 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093586 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093588 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093591 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093593 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093596 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093598 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:07:00.417781 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093601 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093603 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093606 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093608 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093612 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093616 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093618 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093621 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093624 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093626 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093629 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093632 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093634 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093637 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093640 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093643 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093645 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093648 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093651 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093654 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:07:00.418376 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093656 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093660 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093663 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093666 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093668 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093670 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093675 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093677 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093680 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093682 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093685 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093687 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093690 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093693 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.093695 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093770 2559 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093777 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093783 2559 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093787 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093792 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093795 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 12:07:00.418889 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093799 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093803 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093807 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093810 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093813 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093816 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093819 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093822 2559 flags.go:64] FLAG: --cgroup-root="" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093825 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093828 2559 flags.go:64] FLAG: --client-ca-file="" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093832 2559 flags.go:64] FLAG: --cloud-provider="external" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093835 2559 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093842 2559 flags.go:64] FLAG: --cluster-domain="" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093845 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093848 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093851 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093854 2559 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093859 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093862 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093865 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093868 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093871 2559 flags.go:64] FLAG: --contention-profiling="false" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093874 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093877 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 12:07:00.419604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093880 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093883 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093887 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093890 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093893 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093896 2559 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093899 2559 flags.go:64] FLAG: --enable-server="true" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.093902 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094522 2559 flags.go:64] FLAG: --event-burst="100" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094526 2559 flags.go:64] FLAG: --event-qps="50" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094529 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094532 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094535 2559 flags.go:64] FLAG: --eviction-hard="" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094539 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094542 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094545 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094548 2559 flags.go:64] FLAG: --eviction-soft="" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094551 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094554 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094558 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094562 2559 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094565 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094568 2559 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094570 2559 flags.go:64] FLAG: --feature-gates="" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094574 2559 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 12:07:00.420213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094579 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094582 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094585 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094588 2559 flags.go:64] FLAG: --healthz-port="10248" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094590 2559 flags.go:64] FLAG: --help="false" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094593 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-143-101.ec2.internal" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094596 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094599 2559 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094602 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094606 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094609 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094612 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094615 2559 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094618 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094621 2559 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094624 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094627 2559 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094629 2559 flags.go:64] FLAG: --kube-reserved="" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094632 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094635 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094638 2559 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094641 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094644 2559 flags.go:64] FLAG: --lock-file="" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094647 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 12:07:00.420980 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094650 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094653 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094659 2559 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094663 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094667 2559 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094669 2559 flags.go:64] FLAG: --logging-format="text" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094672 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094675 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094680 2559 flags.go:64] FLAG: --manifest-url="" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094683 2559 flags.go:64] FLAG: --manifest-url-header="" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094687 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094691 2559 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094695 2559 flags.go:64] FLAG: --max-pods="110" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094698 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094701 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094704 2559 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094707 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094710 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094713 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094716 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094724 2559 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094727 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094730 2559 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 12:07:00.421637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094733 2559 flags.go:64] FLAG: --pod-cidr="" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094736 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094741 2559 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094744 2559 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094747 2559 flags.go:64] FLAG: --pods-per-core="0" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094750 2559 flags.go:64] FLAG: --port="10250" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094753 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094756 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-091670d3b7d06a95d" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094760 2559 flags.go:64] FLAG: --qos-reserved="" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094763 2559 flags.go:64] FLAG: --read-only-port="10255" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094766 2559 flags.go:64] FLAG: --register-node="true" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094768 2559 flags.go:64] FLAG: --register-with-taints="" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094777 2559 flags.go:64] FLAG: --registry-burst="10" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094782 2559 flags.go:64] FLAG: --registry-qps="5" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094785 2559 flags.go:64] FLAG: --reserved-cpus="" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094788 2559 flags.go:64] FLAG: --reserved-memory="" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094791 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094796 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094799 2559 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094802 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094805 2559 flags.go:64] FLAG: --runonce="false" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094808 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094811 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094814 2559 flags.go:64] FLAG: --seccomp-default="false" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094817 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 12:07:00.422283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094820 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094823 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094826 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094829 2559 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094831 2559 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094834 2559 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094837 2559 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094840 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094843 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094846 2559 flags.go:64] FLAG: --system-cgroups="" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094849 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094854 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094857 2559 flags.go:64] FLAG: --tls-cert-file="" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094860 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094866 2559 flags.go:64] FLAG: --tls-min-version="" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094869 2559 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094872 2559 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094875 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094878 2559 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094881 2559 flags.go:64] FLAG: --v="2" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094886 2559 flags.go:64] FLAG: --version="false" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094891 2559 flags.go:64] FLAG: --vmodule="" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094895 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.094899 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095018 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:07:00.423300 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095021 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095025 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095028 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095031 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095034 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095037 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095040 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095043 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095046 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095049 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095052 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095054 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095057 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095060 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095062 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095065 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095067 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095070 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095072 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095075 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:07:00.423998 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095077 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095080 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095082 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095085 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095088 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095090 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095094 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095099 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095102 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095105 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095110 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095115 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095118 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095120 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095123 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095126 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095128 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095131 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095134 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:07:00.549961 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095136 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095139 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095141 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095144 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095146 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095149 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095151 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095154 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095156 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095159 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095161 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095164 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095167 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095169 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095172 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095175 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095177 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095180 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095182 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095185 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:07:00.550590 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095189 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095191 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095194 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095197 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095202 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095205 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095208 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095210 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095213 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095215 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095218 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095222 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095225 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095228 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095230 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095234 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095236 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095239 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095242 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:07:00.551125 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095245 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095247 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095251 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095253 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095256 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095258 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095261 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095263 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095266 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095268 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095271 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095274 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095276 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095280 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095283 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095286 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095288 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095293 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095295 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:07:00.551682 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.095298 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:07:00.552189 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.096065 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 12:07:00.552189 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.103343 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 12:07:00.552189 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.103357 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 12:07:00.552189 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.103372 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 12:07:00.552189 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.103377 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 12:07:00.552189 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103434 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:07:00.552189 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103439 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:07:00.552189 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103443 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:07:00.552189 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103446 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:07:00.552189 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103449 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:07:00.552189 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103452 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:07:00.552189 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103455 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:07:00.552189 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103458 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:07:00.552189 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103461 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:07:00.552189 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103464 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103467 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103470 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103473 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103476 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103479 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103482 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103484 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103487 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103489 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103492 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103494 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103501 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103506 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103508 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103514 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103516 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103519 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103522 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:07:00.552664 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103525 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103529 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103531 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103534 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103536 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103539 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103542 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103544 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103547 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103549 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103552 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103554 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103557 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103559 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103562 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103564 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103567 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103569 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103572 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103576 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:07:00.553227 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103580 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103583 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103585 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103590 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103593 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103597 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103600 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103603 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103607 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103610 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103612 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103615 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103618 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103620 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103623 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103626 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103628 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103631 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103633 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103636 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:07:00.553802 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103638 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103641 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103644 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103647 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103649 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103652 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103654 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103656 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103659 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103661 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103664 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103666 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103669 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103671 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103674 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103676 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103679 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103683 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103685 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103688 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:07:00.554505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103692 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:07:00.555087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103695 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:07:00.555087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103697 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:07:00.555087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103700 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:07:00.555087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103702 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:07:00.555087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103705 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:07:00.555087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103708 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:07:00.555087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103711 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:07:00.555087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103713 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:07:00.555087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103716 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:07:00.555087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103719 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:07:00.555087 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.103724 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 12:07:00.555087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103832 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:07:00.555087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103837 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:07:00.555087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103840 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:07:00.555087 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103843 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103846 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103849 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103852 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103854 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103857 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103859 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103862 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103864 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103867 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103869 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103872 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103874 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103877 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103879 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103884 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103887 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103891 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103894 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103897 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:07:00.555520 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103899 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103902 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103904 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103907 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103909 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103913 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103918 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103920 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103923 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103925 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103928 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103930 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103933 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103936 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103939 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103941 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103943 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103946 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103948 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103951 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:07:00.556074 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103953 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103956 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103959 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103961 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103964 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103966 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103968 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103973 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103975 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103979 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103983 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103986 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103990 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103993 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103996 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.103999 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104001 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104004 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104007 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:07:00.556619 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104010 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104012 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104016 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104020 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104023 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104026 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104029 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104031 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104034 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104037 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104040 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104042 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104045 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104048 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104051 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104053 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104056 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104058 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104061 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:07:00.557160 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104063 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:07:00.557811 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104067 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:07:00.557811 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104070 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:07:00.557811 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104075 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:07:00.557811 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104077 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:07:00.557811 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104080 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:07:00.557811 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104082 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:07:00.557811 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104085 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:07:00.557811 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104087 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:07:00.557811 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104090 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:07:00.557811 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104092 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:07:00.557811 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104095 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:07:00.557811 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104098 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:07:00.557811 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104101 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:07:00.557811 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104103 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:07:00.557811 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104106 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:07:00.557811 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104108 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:07:00.557811 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:00.104111 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:07:00.558439 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.104116 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 12:07:00.558439 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.104227 2559 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 12:07:00.558439 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.107077 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 12:07:00.558439 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.108848 2559 server.go:1019] "Starting client certificate rotation" Jul 16 12:07:00.558439 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.108954 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 12:07:00.558439 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.108988 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 12:07:00.558439 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.111716 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 12:07:00.558439 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.115091 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 12:07:00.558439 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.134831 2559 log.go:25] "Validated CRI v1 runtime API" Jul 16 12:07:00.558439 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.137196 2559 log.go:25] "Validated CRI v1 image API" Jul 16 12:07:00.558439 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.138005 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 12:07:00.558439 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.140551 2559 fs.go:135] Filesystem UUIDs: map[1aa2515c-6424-4e5c-bbf4-46d60d648058:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e603df75-3df2-4382-bc22-f956b79e939d:/dev/nvme0n1p4] Jul 16 12:07:00.558781 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.140567 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 12:07:00.558885 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.146397 2559 manager.go:216] Machine: {Timestamp:2026-07-16 12:07:00.144662746 +0000 UTC m=+0.391725378 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099607 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22a2094c290dfdbf8389c95bd9d3e1 SystemUUID:ec22a209-4c29-0dfd-bf83-89c95bd9d3e1 BootID:660cccfa-eb7c-4905-9a57-ea8419dc5a48 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3d:98:7e:20:4f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3d:98:7e:20:4f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:cb:9f:b9:49:e2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 12:07:00.558885 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.146542 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 12:07:00.558885 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.146648 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 12:07:00.558885 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.147709 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 12:07:00.558885 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.150054 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 12:07:00.558885 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.150079 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-101.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 12:07:00.558885 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.150900 2559 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.150909 2559 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.150917 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.150931 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.152271 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.157055 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.157178 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.157209 2559 kubelet.go:486] "Attempting to sync node with API server" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.157219 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.157231 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.157240 2559 kubelet.go:398] "Adding apiserver pod source" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.157249 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.157642 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.157657 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.160011 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.164420 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.164449 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.165279 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.165297 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.165306 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.165315 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.165346 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.165355 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.165382 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.165388 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.165395 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.165400 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.165409 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.165416 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.166343 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.166357 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.169543 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-101.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.169728 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.169813 2559 server.go:1268] "Started kubelet" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.169872 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.170070 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.170055 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.170144 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.174030 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 12:07:00.559176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.174954 2559 server.go:310] "Adding debug handlers to kubelet server" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.175992 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.176713 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zfslz" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.176993 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-101.ec2.internal.18c2c3bd4b2c1196 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-101.ec2.internal,UID:ip-10-0-143-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-101.ec2.internal,},FirstTimestamp:2026-07-16 12:07:00.169552278 +0000 UTC m=+0.416614911,LastTimestamp:2026-07-16 12:07:00.169552278 +0000 UTC m=+0.416614911,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-101.ec2.internal,}" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.183161 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.183176 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.183238 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-101.ec2.internal\" not found" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.183881 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.183898 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.183896 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.184013 2559 reconstruct.go:97] "Volume reconstruction finished" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.184022 2559 reconciler.go:29] "Reconciler: start to sync state" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.184215 2559 factory.go:153] Registering CRI-O factory Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.184231 2559 factory.go:223] Registration of the crio container factory successfully Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.184280 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.184288 2559 factory.go:55] Registering systemd factory Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.184296 2559 factory.go:223] Registration of the systemd container factory successfully Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.184315 2559 factory.go:103] Registering Raw factory Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.184345 2559 manager.go:1196] Started watching for new ooms in manager Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.184725 2559 manager.go:349] Starting recovery of all containers Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.185962 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zfslz" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.192968 2559 manager.go:354] Recovery completed Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.195336 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.198619 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.199572 2559 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-101.ec2.internal\" not found" node="ip-10-0-143-101.ec2.internal" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.200519 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.200550 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.200562 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.201089 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.201100 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.201112 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.201117 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.201759 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.201784 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.201796 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:07:00.560479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.203153 2559 policy_none.go:49] "None policy: Start" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.203167 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.203179 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.205127 2559 policy_none.go:47] "Start" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.243685 2559 manager.go:337] "Starting Device Plugin manager" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.243720 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.243730 2559 server.go:87] "Starting device plugin registration server" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.243983 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.243994 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.244095 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.244194 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.244203 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.244895 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.244936 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-101.ec2.internal\" not found" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.344362 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.346704 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.346735 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.346754 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.346786 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-101.ec2.internal" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.363164 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-101.ec2.internal" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.363189 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-101.ec2.internal\": node \"ip-10-0-143-101.ec2.internal\" not found" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.386810 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-101.ec2.internal\" not found" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.451618 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.452993 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.453043 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.453070 2559 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.453111 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.456698 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.490797 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-101.ec2.internal\" not found" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.554027 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-101.ec2.internal"] Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.554120 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.557704 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.557733 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.557747 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.560046 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.560191 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.560231 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.560753 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:07:00.564078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.560777 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:07:00.644672 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.560788 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:07:00.644672 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.560838 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:07:00.644672 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.560865 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:07:00.644672 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.560881 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:07:00.644672 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.563162 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.ec2.internal" Jul 16 12:07:00.644672 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.563187 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:07:00.644672 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.563877 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:07:00.644672 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.563897 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:07:00.644672 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.563911 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:07:00.644672 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.584935 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b7f01a0953299e98dac4176c400f9a5-config\") pod \"kube-apiserver-proxy-ip-10-0-143-101.ec2.internal\" (UID: \"3b7f01a0953299e98dac4176c400f9a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.ec2.internal" Jul 16 12:07:00.644672 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.584963 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40671ae6695f47aabb33292598c63a42-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal\" (UID: \"40671ae6695f47aabb33292598c63a42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal" Jul 16 12:07:00.644672 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.584987 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40671ae6695f47aabb33292598c63a42-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal\" (UID: \"40671ae6695f47aabb33292598c63a42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal" Jul 16 12:07:00.644672 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.590989 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-101.ec2.internal\" not found" Jul 16 12:07:00.644672 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.634371 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-101.ec2.internal\" not found" node="ip-10-0-143-101.ec2.internal" Jul 16 12:07:00.644672 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.638657 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-101.ec2.internal\" not found" node="ip-10-0-143-101.ec2.internal" Jul 16 12:07:00.805366 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.685080 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b7f01a0953299e98dac4176c400f9a5-config\") pod \"kube-apiserver-proxy-ip-10-0-143-101.ec2.internal\" (UID: \"3b7f01a0953299e98dac4176c400f9a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.ec2.internal" Jul 16 12:07:00.805366 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.685109 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40671ae6695f47aabb33292598c63a42-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal\" (UID: \"40671ae6695f47aabb33292598c63a42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal" Jul 16 12:07:00.805366 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.685127 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40671ae6695f47aabb33292598c63a42-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal\" (UID: \"40671ae6695f47aabb33292598c63a42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal" Jul 16 12:07:00.805366 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.685171 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40671ae6695f47aabb33292598c63a42-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal\" (UID: \"40671ae6695f47aabb33292598c63a42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal" Jul 16 12:07:00.805366 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.685182 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b7f01a0953299e98dac4176c400f9a5-config\") pod \"kube-apiserver-proxy-ip-10-0-143-101.ec2.internal\" (UID: \"3b7f01a0953299e98dac4176c400f9a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.ec2.internal" Jul 16 12:07:00.805366 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:00.685214 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40671ae6695f47aabb33292598c63a42-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal\" (UID: \"40671ae6695f47aabb33292598c63a42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal" Jul 16 12:07:00.805366 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.691521 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-101.ec2.internal\" not found" Jul 16 12:07:00.805366 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:00.792061 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-101.ec2.internal\" not found" Jul 16 12:07:03.049523 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:02.992726 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-101.ec2.internal\" not found" Jul 16 12:07:03.049523 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:03.033925 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal" Jul 16 12:07:03.049523 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:03.038889 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.ec2.internal" Jul 16 12:07:03.160610 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:03.092920 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-101.ec2.internal\" not found" Jul 16 12:07:03.164540 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:03.163569 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:07:03.274493 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:03.197728 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-101.ec2.internal\" not found" Jul 16 12:07:03.274493 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:03.204851 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 12:07:03.274493 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:03.204989 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 12:07:03.274493 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:03.205015 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 12:07:03.274493 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:03.205019 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 12:07:03.421705 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:03.289744 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 12:02:02 +0000 UTC" deadline="2028-03-12 17:11:33.84689443 +0000 UTC" Jul 16 12:07:03.421705 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:03.289771 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14525h4m30.557126541s" Jul 16 12:07:03.421705 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:03.297798 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-101.ec2.internal\" not found" Jul 16 12:07:03.421705 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:03.397864 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-101.ec2.internal\" not found" Jul 16 12:07:03.555953 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:03.498423 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-101.ec2.internal\" not found" Jul 16 12:07:03.555953 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:03.528707 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:07:03.704379 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:03.581500 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal" Jul 16 12:07:03.704379 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:03.605854 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 12:07:03.704379 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:03.606673 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.ec2.internal" Jul 16 12:07:03.704379 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:03.619822 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 12:07:03.923515 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:03.914826 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7f01a0953299e98dac4176c400f9a5.slice/crio-9f281cdefd61427616f546372f8fdc619360a0bb27d0255ccfc4d38b3a8642a7 WatchSource:0}: Error finding container 9f281cdefd61427616f546372f8fdc619360a0bb27d0255ccfc4d38b3a8642a7: Status 404 returned error can't find the container with id 9f281cdefd61427616f546372f8fdc619360a0bb27d0255ccfc4d38b3a8642a7 Jul 16 12:07:03.923515 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:03.919736 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:07:04.145943 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.145909 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:07:04.255628 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.255599 2559 apiserver.go:52] "Watching apiserver" Jul 16 12:07:04.263956 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.263891 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 12:07:04.265531 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.265021 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-6wxl5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal","openshift-multus/multus-additional-cni-plugins-pgwg5","openshift-multus/multus-tb25g","openshift-multus/network-metrics-daemon-2wpgm","kube-system/global-pull-secret-syncer-mbwtf","kube-system/kube-apiserver-proxy-ip-10-0-143-101.ec2.internal","openshift-dns/node-resolver-6vr5g","openshift-network-diagnostics/network-check-target-pgxvf","openshift-network-operator/iptables-alerter-gjfsn","openshift-ovn-kubernetes/ovnkube-node-8mwt4","kube-system/konnectivity-agent-f4btg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj","openshift-cluster-node-tuning-operator/tuned-kh5dl"] Jul 16 12:07:04.268935 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.268913 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gjfsn" Jul 16 12:07:04.271018 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.271003 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.272554 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.272533 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:07:04.272648 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.272543 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-66vtx\"" Jul 16 12:07:04.272890 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.272875 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 12:07:04.273002 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.272989 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 12:07:04.273239 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.273221 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.273776 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.273751 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 12:07:04.273776 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.273770 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 12:07:04.273950 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.273751 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 12:07:04.273950 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.273805 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vdv8q\"" Jul 16 12:07:04.275465 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.275363 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.276196 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.276068 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 12:07:04.276196 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.276178 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bxc5v\"" Jul 16 12:07:04.276318 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.276275 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 12:07:04.276389 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.276326 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 12:07:04.277178 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.277162 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 12:07:04.277178 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.277174 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 12:07:04.277627 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.277610 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:04.277741 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:04.277666 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:07:04.278996 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.278978 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 12:07:04.279174 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.279154 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hzn77\"" Jul 16 12:07:04.279825 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.279809 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:04.279902 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:04.279856 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:04.280999 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.280984 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 12:07:04.282032 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.282015 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:04.282123 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:04.282091 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgxvf" podUID="ae76049f-be13-4b92-88b4-9616394c61a5" Jul 16 12:07:04.284292 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.284274 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.286587 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.286525 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6vr5g" Jul 16 12:07:04.287000 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.286981 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:07:04.287369 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.287353 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fggxn\"" Jul 16 12:07:04.287691 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.287662 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 12:07:04.288834 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.288815 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6wxl5" Jul 16 12:07:04.288924 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.288894 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 12:07:04.289000 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.288894 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 12:07:04.289620 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.289608 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jkqj9\"" Jul 16 12:07:04.291491 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.291472 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.292274 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.292249 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 12:07:04.292467 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.292454 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 12:07:04.293495 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.293476 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f4btg" Jul 16 12:07:04.295230 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.295207 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 12:07:04.295360 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.295336 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 12:07:04.295462 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.295364 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 12:07:04.295462 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.295438 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 12:07:04.295604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.295474 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 12:07:04.295604 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.295439 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 12:07:04.296730 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.296519 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dbrpz\"" Jul 16 12:07:04.296730 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.296569 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fc227\"" Jul 16 12:07:04.296730 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.296600 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 12:07:04.296730 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.296519 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 12:07:04.296730 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.296617 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-29t4l\"" Jul 16 12:07:04.296730 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.296649 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 12:07:04.308871 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.308849 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 12:07:04.337239 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.337211 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-778s5" Jul 16 12:07:04.347345 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.347324 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-778s5" Jul 16 12:07:04.362363 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:04.362336 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40671ae6695f47aabb33292598c63a42.slice/crio-f111af05ae2cfd65ca9fa0c920a0246847510a02aef0b7ce150b40f211d2c5a3 WatchSource:0}: Error finding container f111af05ae2cfd65ca9fa0c920a0246847510a02aef0b7ce150b40f211d2c5a3: Status 404 returned error can't find the container with id f111af05ae2cfd65ca9fa0c920a0246847510a02aef0b7ce150b40f211d2c5a3 Jul 16 12:07:04.363051 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363025 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-registration-dir\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.363129 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363065 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-device-dir\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.363129 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363098 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a998397-d529-421e-952b-e0aa50f049f5-cni-binary-copy\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.363129 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363123 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-var-lib-cni-bin\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.363266 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363139 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf48d5a3-b498-4329-a5ec-c0ea64596f64-os-release\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.363266 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363157 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdzjg\" (UniqueName: \"kubernetes.io/projected/d5d90808-177d-4500-b4b2-1ab8bd91003b-kube-api-access-zdzjg\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.363266 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363192 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e31220b-1279-4b66-87c2-ea2948393fd1-dbus\") pod \"global-pull-secret-syncer-mbwtf\" (UID: \"6e31220b-1279-4b66-87c2-ea2948393fd1\") " pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:04.363266 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363207 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret\") pod \"global-pull-secret-syncer-mbwtf\" (UID: \"6e31220b-1279-4b66-87c2-ea2948393fd1\") " pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:04.363266 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363224 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf48d5a3-b498-4329-a5ec-c0ea64596f64-cnibin\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.363266 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363241 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf48d5a3-b498-4329-a5ec-c0ea64596f64-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.363266 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363257 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf48d5a3-b498-4329-a5ec-c0ea64596f64-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.363563 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363273 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b6a1b2d8-b987-4464-89e0-08cda04449bd-iptables-alerter-script\") pod \"iptables-alerter-gjfsn\" (UID: \"b6a1b2d8-b987-4464-89e0-08cda04449bd\") " pod="openshift-network-operator/iptables-alerter-gjfsn" Jul 16 12:07:04.363563 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363287 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6a1b2d8-b987-4464-89e0-08cda04449bd-host-slash\") pod \"iptables-alerter-gjfsn\" (UID: \"b6a1b2d8-b987-4464-89e0-08cda04449bd\") " pod="openshift-network-operator/iptables-alerter-gjfsn" Jul 16 12:07:04.363563 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363302 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-multus-cni-dir\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.363563 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363336 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-cnibin\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.363563 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363357 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-run-k8s-cni-cncf-io\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.363563 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363373 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dfkw\" (UniqueName: \"kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw\") pod \"network-check-target-pgxvf\" (UID: \"ae76049f-be13-4b92-88b4-9616394c61a5\") " pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:04.363563 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363396 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf48d5a3-b498-4329-a5ec-c0ea64596f64-system-cni-dir\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.363563 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363417 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf48d5a3-b498-4329-a5ec-c0ea64596f64-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.363563 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363439 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp6qx\" (UniqueName: \"kubernetes.io/projected/b6a1b2d8-b987-4464-89e0-08cda04449bd-kube-api-access-sp6qx\") pod \"iptables-alerter-gjfsn\" (UID: \"b6a1b2d8-b987-4464-89e0-08cda04449bd\") " pod="openshift-network-operator/iptables-alerter-gjfsn" Jul 16 12:07:04.363563 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363463 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-etc-selinux\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.363563 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363496 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-sys-fs\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.363563 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363550 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-multus-socket-dir-parent\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.364012 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363596 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-run-netns\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.364012 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363613 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-multus-conf-dir\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.364012 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363644 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwqz5\" (UniqueName: \"kubernetes.io/projected/4a998397-d529-421e-952b-e0aa50f049f5-kube-api-access-bwqz5\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.364012 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363665 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs\") pod \"network-metrics-daemon-2wpgm\" (UID: \"d899227b-89c3-4347-8f55-eca200398f00\") " pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:04.364012 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363710 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a998397-d529-421e-952b-e0aa50f049f5-multus-daemon-config\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.364012 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363748 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-socket-dir\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.364012 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363799 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-os-release\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.364012 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363837 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-var-lib-kubelet\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.364012 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363861 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqs4k\" (UniqueName: \"kubernetes.io/projected/d899227b-89c3-4347-8f55-eca200398f00-kube-api-access-kqs4k\") pod \"network-metrics-daemon-2wpgm\" (UID: \"d899227b-89c3-4347-8f55-eca200398f00\") " pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:04.364012 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363903 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e31220b-1279-4b66-87c2-ea2948393fd1-kubelet-config\") pod \"global-pull-secret-syncer-mbwtf\" (UID: \"6e31220b-1279-4b66-87c2-ea2948393fd1\") " pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:04.364012 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363927 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgmj4\" (UniqueName: \"kubernetes.io/projected/cf48d5a3-b498-4329-a5ec-c0ea64596f64-kube-api-access-lgmj4\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.364012 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363943 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-system-cni-dir\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.364012 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.363982 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-var-lib-cni-multus\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.364012 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.364017 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-hostroot\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.364543 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.364042 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-run-multus-certs\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.364543 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.364065 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf48d5a3-b498-4329-a5ec-c0ea64596f64-cni-binary-copy\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.364543 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.364088 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.364543 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.364109 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-etc-kubernetes\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.382236 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.382210 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 12:07:04.464831 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.464796 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-cnibin\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.465013 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.464842 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-run-k8s-cni-cncf-io\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.465013 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.464871 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dfkw\" (UniqueName: \"kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw\") pod \"network-check-target-pgxvf\" (UID: \"ae76049f-be13-4b92-88b4-9616394c61a5\") " pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:04.465013 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.464896 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55mmf\" (UniqueName: \"kubernetes.io/projected/d4b17153-bc06-4588-a2ba-6a11d0de2da9-kube-api-access-55mmf\") pod \"node-ca-6wxl5\" (UID: \"d4b17153-bc06-4588-a2ba-6a11d0de2da9\") " pod="openshift-image-registry/node-ca-6wxl5" Jul 16 12:07:04.465013 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.464914 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-sysctl-d\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.465013 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.464934 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-sys-fs\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.465013 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.464956 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-multus-socket-dir-parent\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.465013 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.464971 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-run-netns\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.465013 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.464987 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwqz5\" (UniqueName: \"kubernetes.io/projected/4a998397-d529-421e-952b-e0aa50f049f5-kube-api-access-bwqz5\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.465013 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465002 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e31220b-1279-4b66-87c2-ea2948393fd1-kubelet-config\") pod \"global-pull-secret-syncer-mbwtf\" (UID: \"6e31220b-1279-4b66-87c2-ea2948393fd1\") " pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:04.465435 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465026 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm55m\" (UniqueName: \"kubernetes.io/projected/d1fe16f9-e90b-4dbf-8462-92f01545a15e-kube-api-access-jm55m\") pod \"node-resolver-6vr5g\" (UID: \"d1fe16f9-e90b-4dbf-8462-92f01545a15e\") " pod="openshift-dns/node-resolver-6vr5g" Jul 16 12:07:04.465435 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465045 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.465435 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465062 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-sys\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.465435 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465080 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-socket-dir\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.465435 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465093 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-hostroot\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.465435 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465114 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf48d5a3-b498-4329-a5ec-c0ea64596f64-cni-binary-copy\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.465435 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465131 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-run\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.465435 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465148 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-etc-openvswitch\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.465435 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465161 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-sysconfig\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.465435 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465180 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d4b17153-bc06-4588-a2ba-6a11d0de2da9-serviceca\") pod \"node-ca-6wxl5\" (UID: \"d4b17153-bc06-4588-a2ba-6a11d0de2da9\") " pod="openshift-image-registry/node-ca-6wxl5" Jul 16 12:07:04.465435 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465195 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-cni-netd\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.465435 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465217 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-device-dir\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.465435 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465239 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a998397-d529-421e-952b-e0aa50f049f5-cni-binary-copy\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.465435 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465301 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-var-lib-cni-bin\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.466169 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465380 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-run-netns\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.466169 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465415 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e31220b-1279-4b66-87c2-ea2948393fd1-kubelet-config\") pod \"global-pull-secret-syncer-mbwtf\" (UID: \"6e31220b-1279-4b66-87c2-ea2948393fd1\") " pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:04.466169 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465464 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-device-dir\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.466169 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465509 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-socket-dir\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.466169 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465511 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-sys-fs\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.466169 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465553 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-cnibin\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.466169 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465590 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-run-k8s-cni-cncf-io\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.466169 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465620 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-node-log\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.466169 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465648 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-log-socket\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.466169 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465671 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b64ae5fe-696a-4fc9-8a0b-249efff10f28-env-overrides\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.466169 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzjg\" (UniqueName: \"kubernetes.io/projected/d5d90808-177d-4500-b4b2-1ab8bd91003b-kube-api-access-zdzjg\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.466169 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465744 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret\") pod \"global-pull-secret-syncer-mbwtf\" (UID: \"6e31220b-1279-4b66-87c2-ea2948393fd1\") " pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:04.466169 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465778 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-lib-modules\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.466169 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465803 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/560b077f-9bc1-4c85-a34a-af83df0daed5-konnectivity-ca\") pod \"konnectivity-agent-f4btg\" (UID: \"560b077f-9bc1-4c85-a34a-af83df0daed5\") " pod="kube-system/konnectivity-agent-f4btg" Jul 16 12:07:04.466801 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465831 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf48d5a3-b498-4329-a5ec-c0ea64596f64-cnibin\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.466801 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465857 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtddt\" (UniqueName: \"kubernetes.io/projected/0f015799-0f53-4767-acb7-d54047f0a4f2-kube-api-access-rtddt\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.466801 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465882 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/560b077f-9bc1-4c85-a34a-af83df0daed5-agent-certs\") pod \"konnectivity-agent-f4btg\" (UID: \"560b077f-9bc1-4c85-a34a-af83df0daed5\") " pod="kube-system/konnectivity-agent-f4btg" Jul 16 12:07:04.466801 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465906 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-multus-cni-dir\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.466801 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465936 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf48d5a3-b498-4329-a5ec-c0ea64596f64-system-cni-dir\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.466801 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.465962 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf48d5a3-b498-4329-a5ec-c0ea64596f64-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.466801 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466008 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a998397-d529-421e-952b-e0aa50f049f5-cni-binary-copy\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.466801 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466076 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-hostroot\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.466801 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466074 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-cni-bin\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.466801 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466113 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb776\" (UniqueName: \"kubernetes.io/projected/b64ae5fe-696a-4fc9-8a0b-249efff10f28-kube-api-access-zb776\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.466801 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466134 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp6qx\" (UniqueName: \"kubernetes.io/projected/b6a1b2d8-b987-4464-89e0-08cda04449bd-kube-api-access-sp6qx\") pod \"iptables-alerter-gjfsn\" (UID: \"b6a1b2d8-b987-4464-89e0-08cda04449bd\") " pod="openshift-network-operator/iptables-alerter-gjfsn" Jul 16 12:07:04.466801 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466151 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-etc-selinux\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.466801 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-multus-conf-dir\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.466801 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466183 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs\") pod \"network-metrics-daemon-2wpgm\" (UID: \"d899227b-89c3-4347-8f55-eca200398f00\") " pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:04.467410 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466213 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-kubelet\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.467410 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466231 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-slash\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.467410 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466253 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-host\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.467410 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466278 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a998397-d529-421e-952b-e0aa50f049f5-multus-daemon-config\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.467410 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466299 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-os-release\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.467410 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466319 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-var-lib-kubelet\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.467410 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466336 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqs4k\" (UniqueName: \"kubernetes.io/projected/d899227b-89c3-4347-8f55-eca200398f00-kube-api-access-kqs4k\") pod \"network-metrics-daemon-2wpgm\" (UID: \"d899227b-89c3-4347-8f55-eca200398f00\") " pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:04.467410 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466361 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgmj4\" (UniqueName: \"kubernetes.io/projected/cf48d5a3-b498-4329-a5ec-c0ea64596f64-kube-api-access-lgmj4\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.467410 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466381 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4b17153-bc06-4588-a2ba-6a11d0de2da9-host\") pod \"node-ca-6wxl5\" (UID: \"d4b17153-bc06-4588-a2ba-6a11d0de2da9\") " pod="openshift-image-registry/node-ca-6wxl5" Jul 16 12:07:04.467410 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466396 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-run-systemd\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.467410 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466422 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-system-cni-dir\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.467410 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466440 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-var-lib-cni-multus\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.467410 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466456 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-run-multus-certs\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.467410 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466483 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-kubernetes\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.467987 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466524 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-run-openvswitch\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.467987 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466548 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-run-ovn\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.467987 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466572 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b64ae5fe-696a-4fc9-8a0b-249efff10f28-ovnkube-config\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.467987 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466575 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf48d5a3-b498-4329-a5ec-c0ea64596f64-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.467987 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466720 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b64ae5fe-696a-4fc9-8a0b-249efff10f28-ovnkube-script-lib\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.467987 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466750 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.467987 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466775 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-etc-kubernetes\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.467987 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466808 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-modprobe-d\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.467987 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466833 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d1fe16f9-e90b-4dbf-8462-92f01545a15e-hosts-file\") pod \"node-resolver-6vr5g\" (UID: \"d1fe16f9-e90b-4dbf-8462-92f01545a15e\") " pod="openshift-dns/node-resolver-6vr5g" Jul 16 12:07:04.467987 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466858 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fe16f9-e90b-4dbf-8462-92f01545a15e-tmp-dir\") pod \"node-resolver-6vr5g\" (UID: \"d1fe16f9-e90b-4dbf-8462-92f01545a15e\") " pod="openshift-dns/node-resolver-6vr5g" Jul 16 12:07:04.467987 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466882 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-systemd-units\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.467987 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466912 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-run-netns\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.467987 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466942 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-registration-dir\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.467987 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466975 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf48d5a3-b498-4329-a5ec-c0ea64596f64-os-release\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.468515 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.466997 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-tuned\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.468515 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467033 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b64ae5fe-696a-4fc9-8a0b-249efff10f28-ovn-node-metrics-cert\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.468515 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467063 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e31220b-1279-4b66-87c2-ea2948393fd1-dbus\") pod \"global-pull-secret-syncer-mbwtf\" (UID: \"6e31220b-1279-4b66-87c2-ea2948393fd1\") " pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:04.468515 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467087 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a998397-d529-421e-952b-e0aa50f049f5-multus-daemon-config\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.468515 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467168 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-os-release\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.468515 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467210 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-var-lib-kubelet\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.468515 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467090 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-systemd\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.468515 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467521 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf48d5a3-b498-4329-a5ec-c0ea64596f64-cni-binary-copy\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.468515 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467527 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf48d5a3-b498-4329-a5ec-c0ea64596f64-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.468515 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467574 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf48d5a3-b498-4329-a5ec-c0ea64596f64-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.468515 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467626 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-var-lib-kubelet\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.468515 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467654 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f015799-0f53-4767-acb7-d54047f0a4f2-tmp\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.468515 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467698 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-var-lib-openvswitch\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.468515 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467707 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf48d5a3-b498-4329-a5ec-c0ea64596f64-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.469167 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467726 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.469167 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467753 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-sysctl-conf\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.469167 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467783 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b6a1b2d8-b987-4464-89e0-08cda04449bd-iptables-alerter-script\") pod \"iptables-alerter-gjfsn\" (UID: \"b6a1b2d8-b987-4464-89e0-08cda04449bd\") " pod="openshift-network-operator/iptables-alerter-gjfsn" Jul 16 12:07:04.469167 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467811 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6a1b2d8-b987-4464-89e0-08cda04449bd-host-slash\") pod \"iptables-alerter-gjfsn\" (UID: \"b6a1b2d8-b987-4464-89e0-08cda04449bd\") " pod="openshift-network-operator/iptables-alerter-gjfsn" Jul 16 12:07:04.469167 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:04.467932 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:04.469167 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467966 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-var-lib-cni-multus\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.469167 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:04.468008 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret podName:6e31220b-1279-4b66-87c2-ea2948393fd1 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:04.967986752 +0000 UTC m=+3.117694165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret") pod "global-pull-secret-syncer-mbwtf" (UID: "6e31220b-1279-4b66-87c2-ea2948393fd1") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:04.469167 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:04.468086 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:04.469167 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.468183 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-multus-cni-dir\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.469167 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:04.468210 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs podName:d899227b-89c3-4347-8f55-eca200398f00 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:04.968188059 +0000 UTC m=+3.117895476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs") pod "network-metrics-daemon-2wpgm" (UID: "d899227b-89c3-4347-8f55-eca200398f00") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:04.469167 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.468227 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf48d5a3-b498-4329-a5ec-c0ea64596f64-system-cni-dir\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.469167 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.468251 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-run-multus-certs\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.469167 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.467903 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-multus-conf-dir\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.469167 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.468294 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-etc-selinux\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.469950 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.468345 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.469950 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.468387 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-etc-kubernetes\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.469950 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.468453 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-host-var-lib-cni-bin\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.469950 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.468496 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf48d5a3-b498-4329-a5ec-c0ea64596f64-os-release\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.469950 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.468538 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5d90808-177d-4500-b4b2-1ab8bd91003b-registration-dir\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.469950 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.468566 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e31220b-1279-4b66-87c2-ea2948393fd1-dbus\") pod \"global-pull-secret-syncer-mbwtf\" (UID: \"6e31220b-1279-4b66-87c2-ea2948393fd1\") " pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:04.469950 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.468580 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-system-cni-dir\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.469950 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.468632 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6a1b2d8-b987-4464-89e0-08cda04449bd-host-slash\") pod \"iptables-alerter-gjfsn\" (UID: \"b6a1b2d8-b987-4464-89e0-08cda04449bd\") " pod="openshift-network-operator/iptables-alerter-gjfsn" Jul 16 12:07:04.469950 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.468793 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf48d5a3-b498-4329-a5ec-c0ea64596f64-cnibin\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.469950 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.468927 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a998397-d529-421e-952b-e0aa50f049f5-multus-socket-dir-parent\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.469950 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.469056 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf48d5a3-b498-4329-a5ec-c0ea64596f64-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.469950 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.469353 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b6a1b2d8-b987-4464-89e0-08cda04449bd-iptables-alerter-script\") pod \"iptables-alerter-gjfsn\" (UID: \"b6a1b2d8-b987-4464-89e0-08cda04449bd\") " pod="openshift-network-operator/iptables-alerter-gjfsn" Jul 16 12:07:04.482911 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.482709 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 12:07:04.484025 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:04.483999 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:07:04.484025 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:04.484023 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:07:04.484187 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:04.484035 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9dfkw for pod openshift-network-diagnostics/network-check-target-pgxvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:04.484187 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:04.484132 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw podName:ae76049f-be13-4b92-88b4-9616394c61a5 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:04.984115718 +0000 UTC m=+3.133823148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9dfkw" (UniqueName: "kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw") pod "network-check-target-pgxvf" (UID: "ae76049f-be13-4b92-88b4-9616394c61a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:04.487078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.487053 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwqz5\" (UniqueName: \"kubernetes.io/projected/4a998397-d529-421e-952b-e0aa50f049f5-kube-api-access-bwqz5\") pod \"multus-tb25g\" (UID: \"4a998397-d529-421e-952b-e0aa50f049f5\") " pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.488524 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.488502 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqs4k\" (UniqueName: \"kubernetes.io/projected/d899227b-89c3-4347-8f55-eca200398f00-kube-api-access-kqs4k\") pod \"network-metrics-daemon-2wpgm\" (UID: \"d899227b-89c3-4347-8f55-eca200398f00\") " pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:04.488788 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.488765 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdzjg\" (UniqueName: \"kubernetes.io/projected/d5d90808-177d-4500-b4b2-1ab8bd91003b-kube-api-access-zdzjg\") pod \"aws-ebs-csi-driver-node-k6dbj\" (UID: \"d5d90808-177d-4500-b4b2-1ab8bd91003b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.489499 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.489457 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgmj4\" (UniqueName: \"kubernetes.io/projected/cf48d5a3-b498-4329-a5ec-c0ea64596f64-kube-api-access-lgmj4\") pod \"multus-additional-cni-plugins-pgwg5\" (UID: \"cf48d5a3-b498-4329-a5ec-c0ea64596f64\") " pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.492566 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.492541 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp6qx\" (UniqueName: \"kubernetes.io/projected/b6a1b2d8-b987-4464-89e0-08cda04449bd-kube-api-access-sp6qx\") pod \"iptables-alerter-gjfsn\" (UID: \"b6a1b2d8-b987-4464-89e0-08cda04449bd\") " pod="openshift-network-operator/iptables-alerter-gjfsn" Jul 16 12:07:04.494105 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.494080 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:07:04.557413 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.557301 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal" event={"ID":"40671ae6695f47aabb33292598c63a42","Type":"ContainerStarted","Data":"f111af05ae2cfd65ca9fa0c920a0246847510a02aef0b7ce150b40f211d2c5a3"} Jul 16 12:07:04.558706 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.558663 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.ec2.internal" event={"ID":"3b7f01a0953299e98dac4176c400f9a5","Type":"ContainerStarted","Data":"9f281cdefd61427616f546372f8fdc619360a0bb27d0255ccfc4d38b3a8642a7"} Jul 16 12:07:04.568370 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568336 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-sysconfig\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.568530 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568382 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d4b17153-bc06-4588-a2ba-6a11d0de2da9-serviceca\") pod \"node-ca-6wxl5\" (UID: \"d4b17153-bc06-4588-a2ba-6a11d0de2da9\") " pod="openshift-image-registry/node-ca-6wxl5" Jul 16 12:07:04.568530 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568399 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-cni-netd\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.568530 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568416 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-node-log\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.568530 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568432 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-log-socket\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.568530 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568438 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-sysconfig\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.568530 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568465 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b64ae5fe-696a-4fc9-8a0b-249efff10f28-env-overrides\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.568530 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568486 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-cni-netd\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.568530 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568502 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-lib-modules\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.568530 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568507 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-node-log\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.568530 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568526 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/560b077f-9bc1-4c85-a34a-af83df0daed5-konnectivity-ca\") pod \"konnectivity-agent-f4btg\" (UID: \"560b077f-9bc1-4c85-a34a-af83df0daed5\") " pod="kube-system/konnectivity-agent-f4btg" Jul 16 12:07:04.569015 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568537 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-log-socket\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.569015 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568553 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtddt\" (UniqueName: \"kubernetes.io/projected/0f015799-0f53-4767-acb7-d54047f0a4f2-kube-api-access-rtddt\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.569015 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568574 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/560b077f-9bc1-4c85-a34a-af83df0daed5-agent-certs\") pod \"konnectivity-agent-f4btg\" (UID: \"560b077f-9bc1-4c85-a34a-af83df0daed5\") " pod="kube-system/konnectivity-agent-f4btg" Jul 16 12:07:04.569015 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568594 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-cni-bin\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.569015 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568612 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb776\" (UniqueName: \"kubernetes.io/projected/b64ae5fe-696a-4fc9-8a0b-249efff10f28-kube-api-access-zb776\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.569015 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568644 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-kubelet\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.569015 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568666 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-slash\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.569015 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568668 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-lib-modules\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.569015 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568705 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-host\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.569015 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568731 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4b17153-bc06-4588-a2ba-6a11d0de2da9-host\") pod \"node-ca-6wxl5\" (UID: \"d4b17153-bc06-4588-a2ba-6a11d0de2da9\") " pod="openshift-image-registry/node-ca-6wxl5" Jul 16 12:07:04.569015 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568753 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-run-systemd\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.569015 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568779 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-kubernetes\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.569015 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568803 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-run-openvswitch\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.569015 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568833 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-run-ovn\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.569015 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568855 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b64ae5fe-696a-4fc9-8a0b-249efff10f28-ovnkube-config\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.569708 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568879 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b64ae5fe-696a-4fc9-8a0b-249efff10f28-ovnkube-script-lib\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.569708 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568902 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-modprobe-d\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.569708 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d1fe16f9-e90b-4dbf-8462-92f01545a15e-hosts-file\") pod \"node-resolver-6vr5g\" (UID: \"d1fe16f9-e90b-4dbf-8462-92f01545a15e\") " pod="openshift-dns/node-resolver-6vr5g" Jul 16 12:07:04.569708 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568970 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fe16f9-e90b-4dbf-8462-92f01545a15e-tmp-dir\") pod \"node-resolver-6vr5g\" (UID: \"d1fe16f9-e90b-4dbf-8462-92f01545a15e\") " pod="openshift-dns/node-resolver-6vr5g" Jul 16 12:07:04.569708 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568987 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b64ae5fe-696a-4fc9-8a0b-249efff10f28-env-overrides\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.569708 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568987 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d4b17153-bc06-4588-a2ba-6a11d0de2da9-serviceca\") pod \"node-ca-6wxl5\" (UID: \"d4b17153-bc06-4588-a2ba-6a11d0de2da9\") " pod="openshift-image-registry/node-ca-6wxl5" Jul 16 12:07:04.569708 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568994 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-systemd-units\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.569708 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569035 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-systemd-units\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.569708 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569054 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-run-netns\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.569708 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569089 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-tuned\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.569708 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569115 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b64ae5fe-696a-4fc9-8a0b-249efff10f28-ovn-node-metrics-cert\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.569708 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569140 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-systemd\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.569708 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-var-lib-kubelet\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.569708 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569189 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f015799-0f53-4767-acb7-d54047f0a4f2-tmp\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.569708 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569211 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-var-lib-openvswitch\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.570405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569237 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.570405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569281 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-sysctl-conf\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.570405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569306 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-run-netns\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.570405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569311 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-host\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.570405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569321 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55mmf\" (UniqueName: \"kubernetes.io/projected/d4b17153-bc06-4588-a2ba-6a11d0de2da9-kube-api-access-55mmf\") pod \"node-ca-6wxl5\" (UID: \"d4b17153-bc06-4588-a2ba-6a11d0de2da9\") " pod="openshift-image-registry/node-ca-6wxl5" Jul 16 12:07:04.570405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569341 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-slash\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.570405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569339 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/560b077f-9bc1-4c85-a34a-af83df0daed5-konnectivity-ca\") pod \"konnectivity-agent-f4btg\" (UID: \"560b077f-9bc1-4c85-a34a-af83df0daed5\") " pod="kube-system/konnectivity-agent-f4btg" Jul 16 12:07:04.570405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.568730 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-cni-bin\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.570405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569386 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-kubelet\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.570405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569392 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-run-systemd\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.570405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569406 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-var-lib-kubelet\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.570405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569412 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.570405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569425 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4b17153-bc06-4588-a2ba-6a11d0de2da9-host\") pod \"node-ca-6wxl5\" (UID: \"d4b17153-bc06-4588-a2ba-6a11d0de2da9\") " pod="openshift-image-registry/node-ca-6wxl5" Jul 16 12:07:04.570405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569447 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-systemd\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.570405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569474 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-sysctl-d\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.571122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569501 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm55m\" (UniqueName: \"kubernetes.io/projected/d1fe16f9-e90b-4dbf-8462-92f01545a15e-kube-api-access-jm55m\") pod \"node-resolver-6vr5g\" (UID: \"d1fe16f9-e90b-4dbf-8462-92f01545a15e\") " pod="openshift-dns/node-resolver-6vr5g" Jul 16 12:07:04.571122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569528 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.571122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569553 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-sys\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.571122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569552 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-sysctl-conf\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.571122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569582 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-run\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.571122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569614 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-var-lib-openvswitch\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.571122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569643 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-etc-openvswitch\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.571122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569615 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-etc-openvswitch\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.571122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569718 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.571122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569731 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-sys\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.571122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569744 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-sysctl-d\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.571122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569915 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-run-openvswitch\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.571122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569921 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-kubernetes\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.571122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569959 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b64ae5fe-696a-4fc9-8a0b-249efff10f28-run-ovn\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.571122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.569976 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-run\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.571893 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.570024 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d1fe16f9-e90b-4dbf-8462-92f01545a15e-hosts-file\") pod \"node-resolver-6vr5g\" (UID: \"d1fe16f9-e90b-4dbf-8462-92f01545a15e\") " pod="openshift-dns/node-resolver-6vr5g" Jul 16 12:07:04.571893 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.570148 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-modprobe-d\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.571893 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.570206 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fe16f9-e90b-4dbf-8462-92f01545a15e-tmp-dir\") pod \"node-resolver-6vr5g\" (UID: \"d1fe16f9-e90b-4dbf-8462-92f01545a15e\") " pod="openshift-dns/node-resolver-6vr5g" Jul 16 12:07:04.571893 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.570410 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b64ae5fe-696a-4fc9-8a0b-249efff10f28-ovnkube-script-lib\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.571893 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.570415 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b64ae5fe-696a-4fc9-8a0b-249efff10f28-ovnkube-config\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.572130 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.572102 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/560b077f-9bc1-4c85-a34a-af83df0daed5-agent-certs\") pod \"konnectivity-agent-f4btg\" (UID: \"560b077f-9bc1-4c85-a34a-af83df0daed5\") " pod="kube-system/konnectivity-agent-f4btg" Jul 16 12:07:04.572874 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.572855 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0f015799-0f53-4767-acb7-d54047f0a4f2-etc-tuned\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.573451 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.573429 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b64ae5fe-696a-4fc9-8a0b-249efff10f28-ovn-node-metrics-cert\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.573657 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.573641 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f015799-0f53-4767-acb7-d54047f0a4f2-tmp\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.579742 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.579714 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gjfsn" Jul 16 12:07:04.584234 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.584142 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" Jul 16 12:07:04.587408 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.587388 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb776\" (UniqueName: \"kubernetes.io/projected/b64ae5fe-696a-4fc9-8a0b-249efff10f28-kube-api-access-zb776\") pod \"ovnkube-node-8mwt4\" (UID: \"b64ae5fe-696a-4fc9-8a0b-249efff10f28\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.588094 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.588072 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtddt\" (UniqueName: \"kubernetes.io/projected/0f015799-0f53-4767-acb7-d54047f0a4f2-kube-api-access-rtddt\") pod \"tuned-kh5dl\" (UID: \"0f015799-0f53-4767-acb7-d54047f0a4f2\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.588558 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.588534 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55mmf\" (UniqueName: \"kubernetes.io/projected/d4b17153-bc06-4588-a2ba-6a11d0de2da9-kube-api-access-55mmf\") pod \"node-ca-6wxl5\" (UID: \"d4b17153-bc06-4588-a2ba-6a11d0de2da9\") " pod="openshift-image-registry/node-ca-6wxl5" Jul 16 12:07:04.589474 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.589454 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm55m\" (UniqueName: \"kubernetes.io/projected/d1fe16f9-e90b-4dbf-8462-92f01545a15e-kube-api-access-jm55m\") pod \"node-resolver-6vr5g\" (UID: \"d1fe16f9-e90b-4dbf-8462-92f01545a15e\") " pod="openshift-dns/node-resolver-6vr5g" Jul 16 12:07:04.591220 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.591197 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pgwg5" Jul 16 12:07:04.594548 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:04.594523 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a1b2d8_b987_4464_89e0_08cda04449bd.slice/crio-404907d6dfeebc808ee716cc845ffd4d50de8496ae1e8a48540294d2ed7ee40c WatchSource:0}: Error finding container 404907d6dfeebc808ee716cc845ffd4d50de8496ae1e8a48540294d2ed7ee40c: Status 404 returned error can't find the container with id 404907d6dfeebc808ee716cc845ffd4d50de8496ae1e8a48540294d2ed7ee40c Jul 16 12:07:04.597553 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.597531 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tb25g" Jul 16 12:07:04.605530 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.605504 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" Jul 16 12:07:04.606505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:04.606473 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf48d5a3_b498_4329_a5ec_c0ea64596f64.slice/crio-6c410bf33302c986b28dc715ff3d39dc51c4f02ca0a8862c54de5a234acfd2e2 WatchSource:0}: Error finding container 6c410bf33302c986b28dc715ff3d39dc51c4f02ca0a8862c54de5a234acfd2e2: Status 404 returned error can't find the container with id 6c410bf33302c986b28dc715ff3d39dc51c4f02ca0a8862c54de5a234acfd2e2 Jul 16 12:07:04.609505 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:04.609476 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a998397_d529_421e_952b_e0aa50f049f5.slice/crio-f9ea5d11fadcaecc375778f686378573f4520bca5370025979c4cf51cf847561 WatchSource:0}: Error finding container f9ea5d11fadcaecc375778f686378573f4520bca5370025979c4cf51cf847561: Status 404 returned error can't find the container with id f9ea5d11fadcaecc375778f686378573f4520bca5370025979c4cf51cf847561 Jul 16 12:07:04.610722 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.610703 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6vr5g" Jul 16 12:07:04.615762 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.615741 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6wxl5" Jul 16 12:07:04.616289 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:04.616266 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f015799_0f53_4767_acb7_d54047f0a4f2.slice/crio-bf7abd22c34463965269d81c453f9d76b11cc3be5679d400343c4c130182f5c6 WatchSource:0}: Error finding container bf7abd22c34463965269d81c453f9d76b11cc3be5679d400343c4c130182f5c6: Status 404 returned error can't find the container with id bf7abd22c34463965269d81c453f9d76b11cc3be5679d400343c4c130182f5c6 Jul 16 12:07:04.621457 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.621434 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:04.621705 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:04.621670 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1fe16f9_e90b_4dbf_8462_92f01545a15e.slice/crio-a1d49dbf197c112d3993abd6bd0572bd0406ac9dc3d19f4c37f382d3e9b90819 WatchSource:0}: Error finding container a1d49dbf197c112d3993abd6bd0572bd0406ac9dc3d19f4c37f382d3e9b90819: Status 404 returned error can't find the container with id a1d49dbf197c112d3993abd6bd0572bd0406ac9dc3d19f4c37f382d3e9b90819 Jul 16 12:07:04.625640 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.625625 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f4btg" Jul 16 12:07:04.626881 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:04.626860 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b17153_bc06_4588_a2ba_6a11d0de2da9.slice/crio-decdbb415a378798408d754a40e761aed28f4fc6147bdd0d0f4a134bcb2509ea WatchSource:0}: Error finding container decdbb415a378798408d754a40e761aed28f4fc6147bdd0d0f4a134bcb2509ea: Status 404 returned error can't find the container with id decdbb415a378798408d754a40e761aed28f4fc6147bdd0d0f4a134bcb2509ea Jul 16 12:07:04.634444 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:04.634418 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64ae5fe_696a_4fc9_8a0b_249efff10f28.slice/crio-94b6badec4a8ecb0888e758d46b8c11285346be8ea603ad76c806e74a3996a74 WatchSource:0}: Error finding container 94b6badec4a8ecb0888e758d46b8c11285346be8ea603ad76c806e74a3996a74: Status 404 returned error can't find the container with id 94b6badec4a8ecb0888e758d46b8c11285346be8ea603ad76c806e74a3996a74 Jul 16 12:07:04.638875 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:04.638850 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560b077f_9bc1_4c85_a34a_af83df0daed5.slice/crio-9f6cd7251296f07655a64b50393c4097da02c583feefcaef6dd97dd0715a20f5 WatchSource:0}: Error finding container 9f6cd7251296f07655a64b50393c4097da02c583feefcaef6dd97dd0715a20f5: Status 404 returned error can't find the container with id 9f6cd7251296f07655a64b50393c4097da02c583feefcaef6dd97dd0715a20f5 Jul 16 12:07:04.971842 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.971761 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret\") pod \"global-pull-secret-syncer-mbwtf\" (UID: \"6e31220b-1279-4b66-87c2-ea2948393fd1\") " pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:04.971842 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:04.971802 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs\") pod \"network-metrics-daemon-2wpgm\" (UID: \"d899227b-89c3-4347-8f55-eca200398f00\") " pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:04.972031 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:04.971925 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:04.972031 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:04.971938 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:04.972031 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:04.971982 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs podName:d899227b-89c3-4347-8f55-eca200398f00 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:05.971967137 +0000 UTC m=+4.121674553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs") pod "network-metrics-daemon-2wpgm" (UID: "d899227b-89c3-4347-8f55-eca200398f00") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:04.972031 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:04.972005 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret podName:6e31220b-1279-4b66-87c2-ea2948393fd1 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:05.97198805 +0000 UTC m=+4.121695483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret") pod "global-pull-secret-syncer-mbwtf" (UID: "6e31220b-1279-4b66-87c2-ea2948393fd1") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:05.072695 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:05.072653 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dfkw\" (UniqueName: \"kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw\") pod \"network-check-target-pgxvf\" (UID: \"ae76049f-be13-4b92-88b4-9616394c61a5\") " pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:05.072871 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:05.072810 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:07:05.072871 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:05.072835 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:07:05.072871 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:05.072845 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9dfkw for pod openshift-network-diagnostics/network-check-target-pgxvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:05.072973 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:05.072897 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw podName:ae76049f-be13-4b92-88b4-9616394c61a5 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:06.072884063 +0000 UTC m=+4.222591487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9dfkw" (UniqueName: "kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw") pod "network-check-target-pgxvf" (UID: "ae76049f-be13-4b92-88b4-9616394c61a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:05.350579 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:05.350481 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 12:02:04 +0000 UTC" deadline="2028-01-25 15:11:15.736743496 +0000 UTC" Jul 16 12:07:05.350579 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:05.350522 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13395h4m10.386225272s" Jul 16 12:07:05.551210 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:05.551178 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:05.551385 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:05.551321 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:07:05.563643 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:05.563602 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" event={"ID":"b64ae5fe-696a-4fc9-8a0b-249efff10f28","Type":"ContainerStarted","Data":"94b6badec4a8ecb0888e758d46b8c11285346be8ea603ad76c806e74a3996a74"} Jul 16 12:07:05.565044 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:05.565015 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6wxl5" event={"ID":"d4b17153-bc06-4588-a2ba-6a11d0de2da9","Type":"ContainerStarted","Data":"decdbb415a378798408d754a40e761aed28f4fc6147bdd0d0f4a134bcb2509ea"} Jul 16 12:07:05.566301 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:05.566277 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6vr5g" event={"ID":"d1fe16f9-e90b-4dbf-8462-92f01545a15e","Type":"ContainerStarted","Data":"a1d49dbf197c112d3993abd6bd0572bd0406ac9dc3d19f4c37f382d3e9b90819"} Jul 16 12:07:05.567792 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:05.567767 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tb25g" event={"ID":"4a998397-d529-421e-952b-e0aa50f049f5","Type":"ContainerStarted","Data":"f9ea5d11fadcaecc375778f686378573f4520bca5370025979c4cf51cf847561"} Jul 16 12:07:05.571813 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:05.571752 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f4btg" event={"ID":"560b077f-9bc1-4c85-a34a-af83df0daed5","Type":"ContainerStarted","Data":"9f6cd7251296f07655a64b50393c4097da02c583feefcaef6dd97dd0715a20f5"} Jul 16 12:07:05.573907 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:05.573887 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" event={"ID":"0f015799-0f53-4767-acb7-d54047f0a4f2","Type":"ContainerStarted","Data":"bf7abd22c34463965269d81c453f9d76b11cc3be5679d400343c4c130182f5c6"} Jul 16 12:07:05.575312 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:05.575283 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgwg5" event={"ID":"cf48d5a3-b498-4329-a5ec-c0ea64596f64","Type":"ContainerStarted","Data":"6c410bf33302c986b28dc715ff3d39dc51c4f02ca0a8862c54de5a234acfd2e2"} Jul 16 12:07:05.577114 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:05.577091 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" event={"ID":"d5d90808-177d-4500-b4b2-1ab8bd91003b","Type":"ContainerStarted","Data":"4160fd5b7a86105e62a8ea025283bab86f7b7d6feb83589eec2390290f8a46de"} Jul 16 12:07:05.578885 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:05.578783 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gjfsn" event={"ID":"b6a1b2d8-b987-4464-89e0-08cda04449bd","Type":"ContainerStarted","Data":"404907d6dfeebc808ee716cc845ffd4d50de8496ae1e8a48540294d2ed7ee40c"} Jul 16 12:07:05.981375 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:05.981331 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs\") pod \"network-metrics-daemon-2wpgm\" (UID: \"d899227b-89c3-4347-8f55-eca200398f00\") " pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:05.981536 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:05.981444 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret\") pod \"global-pull-secret-syncer-mbwtf\" (UID: \"6e31220b-1279-4b66-87c2-ea2948393fd1\") " pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:05.981609 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:05.981574 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:05.981660 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:05.981639 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret podName:6e31220b-1279-4b66-87c2-ea2948393fd1 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:07.981620797 +0000 UTC m=+6.131328211 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret") pod "global-pull-secret-syncer-mbwtf" (UID: "6e31220b-1279-4b66-87c2-ea2948393fd1") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:05.982085 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:05.982048 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:05.982162 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:05.982134 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs podName:d899227b-89c3-4347-8f55-eca200398f00 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:07.982116094 +0000 UTC m=+6.131823510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs") pod "network-metrics-daemon-2wpgm" (UID: "d899227b-89c3-4347-8f55-eca200398f00") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:06.082753 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:06.082611 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dfkw\" (UniqueName: \"kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw\") pod \"network-check-target-pgxvf\" (UID: \"ae76049f-be13-4b92-88b4-9616394c61a5\") " pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:06.082909 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:06.082834 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:07:06.082909 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:06.082856 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:07:06.082909 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:06.082867 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9dfkw for pod openshift-network-diagnostics/network-check-target-pgxvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:06.083056 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:06.082923 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw podName:ae76049f-be13-4b92-88b4-9616394c61a5 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:08.082904428 +0000 UTC m=+6.232611854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9dfkw" (UniqueName: "kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw") pod "network-check-target-pgxvf" (UID: "ae76049f-be13-4b92-88b4-9616394c61a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:06.351718 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:06.351585 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 12:02:04 +0000 UTC" deadline="2028-04-29 23:51:14.147834436 +0000 UTC" Jul 16 12:07:06.351718 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:06.351649 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15683h44m7.796189661s" Jul 16 12:07:06.400630 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:06.400588 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:07:06.551993 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:06.551490 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:06.551993 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:06.551619 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgxvf" podUID="ae76049f-be13-4b92-88b4-9616394c61a5" Jul 16 12:07:06.552464 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:06.552314 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:06.552464 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:06.552421 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:06.598135 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:06.597871 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.ec2.internal" event={"ID":"3b7f01a0953299e98dac4176c400f9a5","Type":"ContainerStarted","Data":"98b18e31daf04251962651db664f397bed7241dd57edf51d8d30151ea3793263"} Jul 16 12:07:07.551449 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:07.551368 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:07.551900 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:07.551516 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:07:07.999898 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:07.999813 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret\") pod \"global-pull-secret-syncer-mbwtf\" (UID: \"6e31220b-1279-4b66-87c2-ea2948393fd1\") " pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:07.999898 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:07.999872 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs\") pod \"network-metrics-daemon-2wpgm\" (UID: \"d899227b-89c3-4347-8f55-eca200398f00\") " pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:08.000133 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:08.000009 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:08.000133 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:08.000057 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:08.000133 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:08.000083 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs podName:d899227b-89c3-4347-8f55-eca200398f00 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:12.000061618 +0000 UTC m=+10.149769057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs") pod "network-metrics-daemon-2wpgm" (UID: "d899227b-89c3-4347-8f55-eca200398f00") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:08.000133 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:08.000124 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret podName:6e31220b-1279-4b66-87c2-ea2948393fd1 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:12.000107856 +0000 UTC m=+10.149815295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret") pod "global-pull-secret-syncer-mbwtf" (UID: "6e31220b-1279-4b66-87c2-ea2948393fd1") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:08.101070 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:08.101035 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dfkw\" (UniqueName: \"kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw\") pod \"network-check-target-pgxvf\" (UID: \"ae76049f-be13-4b92-88b4-9616394c61a5\") " pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:08.101252 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:08.101229 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:07:08.101309 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:08.101262 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:07:08.101309 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:08.101274 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9dfkw for pod openshift-network-diagnostics/network-check-target-pgxvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:08.101400 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:08.101342 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw podName:ae76049f-be13-4b92-88b4-9616394c61a5 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:12.101320321 +0000 UTC m=+10.251027757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9dfkw" (UniqueName: "kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw") pod "network-check-target-pgxvf" (UID: "ae76049f-be13-4b92-88b4-9616394c61a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:08.551430 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:08.550829 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:08.551430 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:08.550970 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:08.551998 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:08.551463 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:08.551998 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:08.551567 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgxvf" podUID="ae76049f-be13-4b92-88b4-9616394c61a5" Jul 16 12:07:09.552014 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:09.551508 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:09.552014 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:09.551660 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:07:10.550958 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:10.550924 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:10.551102 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:10.551064 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:10.551414 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:10.551382 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:10.551519 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:10.551495 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgxvf" podUID="ae76049f-be13-4b92-88b4-9616394c61a5" Jul 16 12:07:11.551276 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:11.551237 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:11.551786 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:11.551404 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:07:12.033896 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:12.033797 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret\") pod \"global-pull-secret-syncer-mbwtf\" (UID: \"6e31220b-1279-4b66-87c2-ea2948393fd1\") " pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:12.033896 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:12.033864 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs\") pod \"network-metrics-daemon-2wpgm\" (UID: \"d899227b-89c3-4347-8f55-eca200398f00\") " pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:12.034126 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:12.033956 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:12.034126 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:12.033986 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:12.034126 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:12.034028 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret podName:6e31220b-1279-4b66-87c2-ea2948393fd1 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:20.034007717 +0000 UTC m=+18.183715144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret") pod "global-pull-secret-syncer-mbwtf" (UID: "6e31220b-1279-4b66-87c2-ea2948393fd1") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:12.034126 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:12.034047 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs podName:d899227b-89c3-4347-8f55-eca200398f00 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:20.034038542 +0000 UTC m=+18.183745961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs") pod "network-metrics-daemon-2wpgm" (UID: "d899227b-89c3-4347-8f55-eca200398f00") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:12.135229 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:12.135180 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dfkw\" (UniqueName: \"kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw\") pod \"network-check-target-pgxvf\" (UID: \"ae76049f-be13-4b92-88b4-9616394c61a5\") " pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:12.135418 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:12.135366 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:07:12.135418 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:12.135385 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:07:12.135418 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:12.135398 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9dfkw for pod openshift-network-diagnostics/network-check-target-pgxvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:12.135567 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:12.135457 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw podName:ae76049f-be13-4b92-88b4-9616394c61a5 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:20.135437849 +0000 UTC m=+18.285145263 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9dfkw" (UniqueName: "kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw") pod "network-check-target-pgxvf" (UID: "ae76049f-be13-4b92-88b4-9616394c61a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:12.551765 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:12.551725 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:12.552250 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:12.551830 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:12.552323 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:12.552289 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:12.552411 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:12.552388 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgxvf" podUID="ae76049f-be13-4b92-88b4-9616394c61a5" Jul 16 12:07:13.551375 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:13.551103 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:13.551375 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:13.551254 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:07:14.551648 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:14.551610 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:14.552162 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:14.551768 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:14.552162 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:14.551828 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:14.552162 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:14.551946 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgxvf" podUID="ae76049f-be13-4b92-88b4-9616394c61a5" Jul 16 12:07:15.550982 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:15.550946 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:15.551122 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:15.551057 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:07:16.550941 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:16.550910 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:16.551416 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:16.550910 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:16.551416 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:16.551042 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:16.551416 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:16.551112 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgxvf" podUID="ae76049f-be13-4b92-88b4-9616394c61a5" Jul 16 12:07:17.561541 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:17.561506 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:17.562025 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:17.561624 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:07:18.551442 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:18.551408 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:18.551633 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:18.551408 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:18.551633 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:18.551550 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:18.551765 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:18.551624 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgxvf" podUID="ae76049f-be13-4b92-88b4-9616394c61a5" Jul 16 12:07:19.551256 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:19.551221 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:19.551696 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:19.551333 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:07:20.091690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:20.091630 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret\") pod \"global-pull-secret-syncer-mbwtf\" (UID: \"6e31220b-1279-4b66-87c2-ea2948393fd1\") " pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:20.091887 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:20.091714 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs\") pod \"network-metrics-daemon-2wpgm\" (UID: \"d899227b-89c3-4347-8f55-eca200398f00\") " pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:20.091887 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:20.091780 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:20.091887 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:20.091868 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:20.092049 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:20.091873 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret podName:6e31220b-1279-4b66-87c2-ea2948393fd1 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:36.091848411 +0000 UTC m=+34.241555845 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret") pod "global-pull-secret-syncer-mbwtf" (UID: "6e31220b-1279-4b66-87c2-ea2948393fd1") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:20.092049 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:20.091936 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs podName:d899227b-89c3-4347-8f55-eca200398f00 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:36.09192088 +0000 UTC m=+34.241628306 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs") pod "network-metrics-daemon-2wpgm" (UID: "d899227b-89c3-4347-8f55-eca200398f00") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:20.192803 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:20.192766 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dfkw\" (UniqueName: \"kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw\") pod \"network-check-target-pgxvf\" (UID: \"ae76049f-be13-4b92-88b4-9616394c61a5\") " pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:20.192980 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:20.192957 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:07:20.193038 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:20.192986 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:07:20.193038 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:20.193000 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9dfkw for pod openshift-network-diagnostics/network-check-target-pgxvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:20.193122 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:20.193064 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw podName:ae76049f-be13-4b92-88b4-9616394c61a5 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:36.193044249 +0000 UTC m=+34.342751661 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9dfkw" (UniqueName: "kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw") pod "network-check-target-pgxvf" (UID: "ae76049f-be13-4b92-88b4-9616394c61a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:20.551419 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:20.551328 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:20.551874 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:20.551338 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:20.551874 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:20.551470 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:20.551874 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:20.551547 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgxvf" podUID="ae76049f-be13-4b92-88b4-9616394c61a5" Jul 16 12:07:21.551109 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:21.551072 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:21.551303 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:21.551218 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:07:22.551620 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:22.551563 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:22.552092 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:22.551695 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:22.552092 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:22.552012 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:22.552212 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:22.552122 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgxvf" podUID="ae76049f-be13-4b92-88b4-9616394c61a5" Jul 16 12:07:23.551162 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:23.551126 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:23.551330 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:23.551268 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:07:24.551499 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:24.551468 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:24.551880 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:24.551468 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:24.551880 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:24.551592 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgxvf" podUID="ae76049f-be13-4b92-88b4-9616394c61a5" Jul 16 12:07:24.551880 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:24.551632 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:24.635292 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:24.635226 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tb25g" event={"ID":"4a998397-d529-421e-952b-e0aa50f049f5","Type":"ContainerStarted","Data":"d511bcd666c8a249d797e77910b375d8fbf7e58ccd4449175ac49deaad39ed02"} Jul 16 12:07:24.638871 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:24.638804 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" event={"ID":"0f015799-0f53-4767-acb7-d54047f0a4f2","Type":"ContainerStarted","Data":"9b8fca8712c3222604574f292b7d2af79e4087ff506844a31e2c7fc3a4599805"} Jul 16 12:07:24.674110 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:24.674023 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.ec2.internal" podStartSLOduration=21.67398746 podStartE2EDuration="21.67398746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:06.618844719 +0000 UTC m=+4.768552153" watchObservedRunningTime="2026-07-16 12:07:24.67398746 +0000 UTC m=+22.823694890" Jul 16 12:07:24.674879 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:24.674841 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tb25g" podStartSLOduration=3.07392491 podStartE2EDuration="22.674825565s" totalImagesPullingTime="19.600900655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:24.67453344 +0000 UTC m=+22.824240873" watchObservedRunningTime="2026-07-16 12:07:24.674825565 +0000 UTC m=+22.824532999" Jul 16 12:07:25.551988 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.551661 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:25.552787 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:25.552116 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:07:25.643250 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.643214 2559 generic.go:356] "Generic (PLEG): container finished" podID="cf48d5a3-b498-4329-a5ec-c0ea64596f64" containerID="4b2235753bb5b731f0c5b638bad32a062f8945a5806bc515b2a5b6ae717f265e" exitCode=0 Jul 16 12:07:25.643429 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.643291 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgwg5" event={"ID":"cf48d5a3-b498-4329-a5ec-c0ea64596f64","Type":"ContainerDied","Data":"4b2235753bb5b731f0c5b638bad32a062f8945a5806bc515b2a5b6ae717f265e"} Jul 16 12:07:25.645153 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.645129 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" event={"ID":"d5d90808-177d-4500-b4b2-1ab8bd91003b","Type":"ContainerStarted","Data":"b0fade6c81589d25f460df03cbaffbd02192ddf8a15a9398d209c191901f66aa"} Jul 16 12:07:25.646852 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.646824 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gjfsn" event={"ID":"b6a1b2d8-b987-4464-89e0-08cda04449bd","Type":"ContainerStarted","Data":"4fe88d3887ebda2648913665c15fb767c766f955a054c93cb3162c197b38444d"} Jul 16 12:07:25.650225 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.650188 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:07:25.650632 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.650609 2559 generic.go:356] "Generic (PLEG): container finished" podID="b64ae5fe-696a-4fc9-8a0b-249efff10f28" containerID="dbfff11ce7a185101a71ed8c62ae9bdd6be3fc2bb63d848a00ee8e507e931a58" exitCode=1 Jul 16 12:07:25.650762 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.650655 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" event={"ID":"b64ae5fe-696a-4fc9-8a0b-249efff10f28","Type":"ContainerStarted","Data":"9a56f528a381280e03fff0c7e6b3bc12f7294b20dc9989bef518b78d17b8768e"} Jul 16 12:07:25.650762 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.650753 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" event={"ID":"b64ae5fe-696a-4fc9-8a0b-249efff10f28","Type":"ContainerStarted","Data":"9ee197056b63c64e285069cfb2bc30827c296b7c683d20a04e0356817bbca1ea"} Jul 16 12:07:25.650900 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.650864 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" event={"ID":"b64ae5fe-696a-4fc9-8a0b-249efff10f28","Type":"ContainerStarted","Data":"79fbab8ff88ff611ba8197b998ae55c0bb41087f654fd7bd09876af87336b9d3"} Jul 16 12:07:25.650959 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.650917 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" event={"ID":"b64ae5fe-696a-4fc9-8a0b-249efff10f28","Type":"ContainerStarted","Data":"398fa3322a0a65d8146c175abd99649d164ead87d06f0bce8464571038b7b895"} Jul 16 12:07:25.651042 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.651016 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" event={"ID":"b64ae5fe-696a-4fc9-8a0b-249efff10f28","Type":"ContainerDied","Data":"dbfff11ce7a185101a71ed8c62ae9bdd6be3fc2bb63d848a00ee8e507e931a58"} Jul 16 12:07:25.651150 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.651072 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" event={"ID":"b64ae5fe-696a-4fc9-8a0b-249efff10f28","Type":"ContainerStarted","Data":"1300c498d0a36ae79ff79db8002624800b12793fd44f4538da9ddf1455baa2fa"} Jul 16 12:07:25.652445 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.652418 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6wxl5" event={"ID":"d4b17153-bc06-4588-a2ba-6a11d0de2da9","Type":"ContainerStarted","Data":"cc45eafb471c0fc50a7bf602a6fd1f7c779091e4f60835a7ca19bece65f5da7e"} Jul 16 12:07:25.654478 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.654447 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6vr5g" event={"ID":"d1fe16f9-e90b-4dbf-8462-92f01545a15e","Type":"ContainerStarted","Data":"1affbc5e547bbd6eb66edc95ff85801276ae83e25a5132fe905ea283aad194d1"} Jul 16 12:07:25.656078 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.656058 2559 generic.go:356] "Generic (PLEG): container finished" podID="40671ae6695f47aabb33292598c63a42" containerID="df565ca7a6f439b614c5c7ab5fc790a948a8ec369ce59fda59d459bd2aed8cea" exitCode=0 Jul 16 12:07:25.656172 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.656103 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal" event={"ID":"40671ae6695f47aabb33292598c63a42","Type":"ContainerDied","Data":"df565ca7a6f439b614c5c7ab5fc790a948a8ec369ce59fda59d459bd2aed8cea"} Jul 16 12:07:25.657713 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.657665 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f4btg" event={"ID":"560b077f-9bc1-4c85-a34a-af83df0daed5","Type":"ContainerStarted","Data":"43e803b359b014aea3a4c66529126293e2e5e354f02088eee8e319d71a5a8fdb"} Jul 16 12:07:25.710318 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.710213 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kh5dl" podStartSLOduration=4.143996547 podStartE2EDuration="23.710197816s" totalImagesPullingTime="19.566201269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:24.708005032 +0000 UTC m=+22.857712464" watchObservedRunningTime="2026-07-16 12:07:25.710197816 +0000 UTC m=+23.859905248" Jul 16 12:07:25.776084 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.776025 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6wxl5" podStartSLOduration=4.220828992 podStartE2EDuration="23.776002293s" totalImagesPullingTime="19.555173301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:25.745267768 +0000 UTC m=+23.894975201" watchObservedRunningTime="2026-07-16 12:07:25.776002293 +0000 UTC m=+23.925709725" Jul 16 12:07:25.805809 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.805764 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f4btg" podStartSLOduration=4.366452017 podStartE2EDuration="23.805746511s" totalImagesPullingTime="19.439294494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:25.776206652 +0000 UTC m=+23.925914084" watchObservedRunningTime="2026-07-16 12:07:25.805746511 +0000 UTC m=+23.955453944" Jul 16 12:07:25.887318 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.887271 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6vr5g" podStartSLOduration=4.326624354 podStartE2EDuration="23.887254615s" totalImagesPullingTime="19.560630261s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:25.88663154 +0000 UTC m=+24.036338973" watchObservedRunningTime="2026-07-16 12:07:25.887254615 +0000 UTC m=+24.036962048" Jul 16 12:07:25.887461 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:25.887374 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gjfsn" podStartSLOduration=4.753152286 podStartE2EDuration="23.887369153s" totalImagesPullingTime="19.134216867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:25.840108554 +0000 UTC m=+23.989815986" watchObservedRunningTime="2026-07-16 12:07:25.887369153 +0000 UTC m=+24.037076585" Jul 16 12:07:26.087883 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:26.087852 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 12:07:26.356992 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:26.356894 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T12:07:26.087878908Z","UUID":"f1245532-d95f-43b7-88bc-2c9e05ad5953","Handler":null,"Name":"","Endpoint":""} Jul 16 12:07:26.359106 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:26.359081 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 12:07:26.359234 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:26.359127 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 12:07:26.551349 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:26.551313 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:26.551553 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:26.551441 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:26.551553 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:26.551475 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:26.551553 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:26.551546 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgxvf" podUID="ae76049f-be13-4b92-88b4-9616394c61a5" Jul 16 12:07:26.661729 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:26.661618 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" event={"ID":"d5d90808-177d-4500-b4b2-1ab8bd91003b","Type":"ContainerStarted","Data":"3a363ba97fc55f15897fb856eb8023ce0f77a63bc792150b602691e8e23e1045"} Jul 16 12:07:26.663481 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:26.663447 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal" event={"ID":"40671ae6695f47aabb33292598c63a42","Type":"ContainerStarted","Data":"56e37d51d376532eb2253814696b16dcf51dbe530f5f54d5f86dc015c8aa7fb4"} Jul 16 12:07:26.785312 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:26.785278 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f4btg" Jul 16 12:07:27.551155 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:27.550947 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:27.551295 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:27.551270 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:07:27.670411 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:27.670374 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" event={"ID":"d5d90808-177d-4500-b4b2-1ab8bd91003b","Type":"ContainerStarted","Data":"4b4dc1974f8ba57bdb785236af24c485824b7ecf00db2621f99202fd094a4079"} Jul 16 12:07:27.749797 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:27.749736 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.ec2.internal" podStartSLOduration=24.749715385000002 podStartE2EDuration="24.749715385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:26.703568062 +0000 UTC m=+24.853275495" watchObservedRunningTime="2026-07-16 12:07:27.749715385 +0000 UTC m=+25.899422819" Jul 16 12:07:28.551357 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:28.551322 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:28.551552 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:28.551320 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:28.551552 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:28.551433 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgxvf" podUID="ae76049f-be13-4b92-88b4-9616394c61a5" Jul 16 12:07:28.551552 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:28.551513 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:28.677177 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:28.677151 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:07:28.677741 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:28.677637 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" event={"ID":"b64ae5fe-696a-4fc9-8a0b-249efff10f28","Type":"ContainerStarted","Data":"b400029722b2edd150ce4bbd3a5f3ce28e71c8238532a1a91164806c1c845c19"} Jul 16 12:07:29.199506 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:29.199470 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f4btg" Jul 16 12:07:29.200364 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:29.200338 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f4btg" Jul 16 12:07:29.239992 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:29.239945 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k6dbj" podStartSLOduration=5.17923349 podStartE2EDuration="27.239927619s" totalImagesPullingTime="22.060694129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:07:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:27.749991708 +0000 UTC m=+25.899699141" watchObservedRunningTime="2026-07-16 12:07:29.239927619 +0000 UTC m=+27.389635053" Jul 16 12:07:29.550882 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:29.550847 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:29.551046 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:29.550993 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:07:29.681024 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:29.680997 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f4btg" Jul 16 12:07:30.550693 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:30.550643 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:30.550693 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:30.550643 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:30.550928 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:30.550790 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgxvf" podUID="ae76049f-be13-4b92-88b4-9616394c61a5" Jul 16 12:07:30.550928 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:30.550909 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:30.686623 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:30.686450 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:07:30.687272 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:30.686971 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" event={"ID":"b64ae5fe-696a-4fc9-8a0b-249efff10f28","Type":"ContainerStarted","Data":"e873170a6035b28db97d1010df77257264e5cdfb96d1551890a1226efa4238ca"} Jul 16 12:07:30.687448 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:30.687429 2559 scope.go:117] "RemoveContainer" containerID="dbfff11ce7a185101a71ed8c62ae9bdd6be3fc2bb63d848a00ee8e507e931a58" Jul 16 12:07:31.551690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:31.551640 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:31.551874 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:31.551777 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:07:31.690559 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:31.690522 2559 generic.go:356] "Generic (PLEG): container finished" podID="cf48d5a3-b498-4329-a5ec-c0ea64596f64" containerID="3e8f86fe4821c568056a1566ae43969b5fdd7c9f59adf9c08d2017e94d9401f6" exitCode=0 Jul 16 12:07:31.690975 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:31.690605 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgwg5" event={"ID":"cf48d5a3-b498-4329-a5ec-c0ea64596f64","Type":"ContainerDied","Data":"3e8f86fe4821c568056a1566ae43969b5fdd7c9f59adf9c08d2017e94d9401f6"} Jul 16 12:07:31.694562 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:31.694539 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:07:31.694875 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:31.694854 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" event={"ID":"b64ae5fe-696a-4fc9-8a0b-249efff10f28","Type":"ContainerStarted","Data":"8b4d66f908c0e27b07eb964b90a3561988e9d5abdd3c359e84f10596e40f8faf"} Jul 16 12:07:31.695175 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:31.695150 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:31.695261 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:31.695222 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:31.695303 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:31.695270 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:31.710882 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:31.710854 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:31.711022 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:31.710957 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:07:32.552433 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:32.552398 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:32.552624 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:32.552441 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:32.552624 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:32.552528 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:32.552751 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:32.552622 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgxvf" podUID="ae76049f-be13-4b92-88b4-9616394c61a5" Jul 16 12:07:33.345139 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:33.344889 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" podStartSLOduration=11.255052635 podStartE2EDuration="31.344869436s" totalImagesPullingTime="20.089816801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:31.805587179 +0000 UTC m=+29.955294612" watchObservedRunningTime="2026-07-16 12:07:33.344869436 +0000 UTC m=+31.494576899" Jul 16 12:07:33.345843 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:33.345491 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2wpgm"] Jul 16 12:07:33.345843 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:33.345608 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:33.345843 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:33.345748 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:07:33.351667 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:33.351643 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mbwtf"] Jul 16 12:07:33.351922 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:33.351772 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:33.351922 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:33.351866 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:33.357551 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:33.355042 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pgxvf"] Jul 16 12:07:33.357551 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:33.355190 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:33.357551 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:33.355300 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgxvf" podUID="ae76049f-be13-4b92-88b4-9616394c61a5" Jul 16 12:07:33.701421 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:33.701322 2559 generic.go:356] "Generic (PLEG): container finished" podID="cf48d5a3-b498-4329-a5ec-c0ea64596f64" containerID="61cf0466ac688127e2b8250b7c6ff8c25489d6771371baf9c809213afd0c4b90" exitCode=0 Jul 16 12:07:33.701421 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:33.701398 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgwg5" event={"ID":"cf48d5a3-b498-4329-a5ec-c0ea64596f64","Type":"ContainerDied","Data":"61cf0466ac688127e2b8250b7c6ff8c25489d6771371baf9c809213afd0c4b90"} Jul 16 12:07:34.551294 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:34.551265 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:34.551703 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:34.551408 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:34.705554 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:34.705522 2559 generic.go:356] "Generic (PLEG): container finished" podID="cf48d5a3-b498-4329-a5ec-c0ea64596f64" containerID="9c4686e21a4e4cb1f5401e7ee9cee78f5da3b23a0bcd58b0f5b584cefaa96183" exitCode=0 Jul 16 12:07:34.705734 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:34.705579 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgwg5" event={"ID":"cf48d5a3-b498-4329-a5ec-c0ea64596f64","Type":"ContainerDied","Data":"9c4686e21a4e4cb1f5401e7ee9cee78f5da3b23a0bcd58b0f5b584cefaa96183"} Jul 16 12:07:35.551165 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:35.551135 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:35.551362 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:35.551147 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:35.551362 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:35.551245 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgxvf" podUID="ae76049f-be13-4b92-88b4-9616394c61a5" Jul 16 12:07:35.551362 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:35.551341 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:07:36.106544 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:36.106509 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret\") pod \"global-pull-secret-syncer-mbwtf\" (UID: \"6e31220b-1279-4b66-87c2-ea2948393fd1\") " pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:36.106544 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:36.106547 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs\") pod \"network-metrics-daemon-2wpgm\" (UID: \"d899227b-89c3-4347-8f55-eca200398f00\") " pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:36.106786 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:36.106644 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:36.106786 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:36.106661 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:36.106786 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:36.106714 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs podName:d899227b-89c3-4347-8f55-eca200398f00 nodeName:}" failed. No retries permitted until 2026-07-16 12:08:08.106699152 +0000 UTC m=+66.256406563 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs") pod "network-metrics-daemon-2wpgm" (UID: "d899227b-89c3-4347-8f55-eca200398f00") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:36.109466 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:36.107131 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret podName:6e31220b-1279-4b66-87c2-ea2948393fd1 nodeName:}" failed. No retries permitted until 2026-07-16 12:08:08.106721879 +0000 UTC m=+66.256429290 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret") pod "global-pull-secret-syncer-mbwtf" (UID: "6e31220b-1279-4b66-87c2-ea2948393fd1") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:36.207644 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:36.207607 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dfkw\" (UniqueName: \"kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw\") pod \"network-check-target-pgxvf\" (UID: \"ae76049f-be13-4b92-88b4-9616394c61a5\") " pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:36.207827 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:36.207797 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:07:36.207827 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:36.207822 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:07:36.207925 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:36.207833 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9dfkw for pod openshift-network-diagnostics/network-check-target-pgxvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:36.207925 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:36.207886 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw podName:ae76049f-be13-4b92-88b4-9616394c61a5 nodeName:}" failed. No retries permitted until 2026-07-16 12:08:08.207870256 +0000 UTC m=+66.357577668 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9dfkw" (UniqueName: "kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw") pod "network-check-target-pgxvf" (UID: "ae76049f-be13-4b92-88b4-9616394c61a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:36.551337 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:36.551304 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:36.551485 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:36.551408 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbwtf" podUID="6e31220b-1279-4b66-87c2-ea2948393fd1" Jul 16 12:07:37.165404 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.165221 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-101.ec2.internal" event="NodeReady" Jul 16 12:07:37.165589 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.165491 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 12:07:37.272569 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.272538 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-f55c4"] Jul 16 12:07:37.275598 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.275575 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f55c4" Jul 16 12:07:37.277640 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.277616 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r4f95"] Jul 16 12:07:37.280304 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.280258 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r4f95" Jul 16 12:07:37.280981 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.280964 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v8966\"" Jul 16 12:07:37.281081 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.281009 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 12:07:37.286331 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.286314 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 12:07:37.286410 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.286365 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 12:07:37.286471 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.286457 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 12:07:37.286573 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.286560 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pwb6c\"" Jul 16 12:07:37.290258 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.290242 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 12:07:37.315795 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.315759 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:07:37.315795 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.315798 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/268f88f9-d545-4d19-9bc8-8695df1dc92c-config-volume\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:07:37.316001 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.315816 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx9kn\" (UniqueName: \"kubernetes.io/projected/268f88f9-d545-4d19-9bc8-8695df1dc92c-kube-api-access-nx9kn\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:07:37.316001 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.315931 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/268f88f9-d545-4d19-9bc8-8695df1dc92c-tmp-dir\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:07:37.326031 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.325998 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r4f95"] Jul 16 12:07:37.329897 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.329864 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f55c4"] Jul 16 12:07:37.416447 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.416406 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/268f88f9-d545-4d19-9bc8-8695df1dc92c-tmp-dir\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:07:37.416625 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.416460 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:07:37.416625 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.416481 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/268f88f9-d545-4d19-9bc8-8695df1dc92c-config-volume\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:07:37.416625 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.416504 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx9kn\" (UniqueName: \"kubernetes.io/projected/268f88f9-d545-4d19-9bc8-8695df1dc92c-kube-api-access-nx9kn\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:07:37.416625 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.416554 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94335d8a-acdd-4739-8741-ce365e66d8c5-cert\") pod \"ingress-canary-r4f95\" (UID: \"94335d8a-acdd-4739-8741-ce365e66d8c5\") " pod="openshift-ingress-canary/ingress-canary-r4f95" Jul 16 12:07:37.416625 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:37.416578 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:07:37.416864 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.416583 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqh24\" (UniqueName: \"kubernetes.io/projected/94335d8a-acdd-4739-8741-ce365e66d8c5-kube-api-access-jqh24\") pod \"ingress-canary-r4f95\" (UID: \"94335d8a-acdd-4739-8741-ce365e66d8c5\") " pod="openshift-ingress-canary/ingress-canary-r4f95" Jul 16 12:07:37.416864 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:37.416663 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls podName:268f88f9-d545-4d19-9bc8-8695df1dc92c nodeName:}" failed. No retries permitted until 2026-07-16 12:07:37.916641734 +0000 UTC m=+36.066349154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls") pod "dns-default-f55c4" (UID: "268f88f9-d545-4d19-9bc8-8695df1dc92c") : secret "dns-default-metrics-tls" not found Jul 16 12:07:37.416864 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.416827 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/268f88f9-d545-4d19-9bc8-8695df1dc92c-tmp-dir\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:07:37.417066 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.417049 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/268f88f9-d545-4d19-9bc8-8695df1dc92c-config-volume\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:07:37.434058 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.434028 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx9kn\" (UniqueName: \"kubernetes.io/projected/268f88f9-d545-4d19-9bc8-8695df1dc92c-kube-api-access-nx9kn\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:07:37.517959 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.517920 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94335d8a-acdd-4739-8741-ce365e66d8c5-cert\") pod \"ingress-canary-r4f95\" (UID: \"94335d8a-acdd-4739-8741-ce365e66d8c5\") " pod="openshift-ingress-canary/ingress-canary-r4f95" Jul 16 12:07:37.517959 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.517961 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqh24\" (UniqueName: \"kubernetes.io/projected/94335d8a-acdd-4739-8741-ce365e66d8c5-kube-api-access-jqh24\") pod \"ingress-canary-r4f95\" (UID: \"94335d8a-acdd-4739-8741-ce365e66d8c5\") " pod="openshift-ingress-canary/ingress-canary-r4f95" Jul 16 12:07:37.520324 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.520295 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94335d8a-acdd-4739-8741-ce365e66d8c5-cert\") pod \"ingress-canary-r4f95\" (UID: \"94335d8a-acdd-4739-8741-ce365e66d8c5\") " pod="openshift-ingress-canary/ingress-canary-r4f95" Jul 16 12:07:37.536023 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.535941 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqh24\" (UniqueName: \"kubernetes.io/projected/94335d8a-acdd-4739-8741-ce365e66d8c5-kube-api-access-jqh24\") pod \"ingress-canary-r4f95\" (UID: \"94335d8a-acdd-4739-8741-ce365e66d8c5\") " pod="openshift-ingress-canary/ingress-canary-r4f95" Jul 16 12:07:37.551672 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.551641 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:07:37.552091 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.551642 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:07:37.559904 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.559881 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 12:07:37.564646 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.564619 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 12:07:37.565165 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.565006 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hz6m9\"" Jul 16 12:07:37.565657 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.565638 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 12:07:37.566519 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.566501 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w29rm\"" Jul 16 12:07:37.591156 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.591113 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r4f95" Jul 16 12:07:37.760849 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.760810 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r4f95"] Jul 16 12:07:37.767046 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:07:37.767015 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94335d8a_acdd_4739_8741_ce365e66d8c5.slice/crio-ffd6f94a6c8cb8fe231cb3691db3b866b4239731d823036b6e60becc761728ea WatchSource:0}: Error finding container ffd6f94a6c8cb8fe231cb3691db3b866b4239731d823036b6e60becc761728ea: Status 404 returned error can't find the container with id ffd6f94a6c8cb8fe231cb3691db3b866b4239731d823036b6e60becc761728ea Jul 16 12:07:37.922223 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:37.922127 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:07:37.922394 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:37.922253 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:07:37.922394 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:37.922334 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls podName:268f88f9-d545-4d19-9bc8-8695df1dc92c nodeName:}" failed. No retries permitted until 2026-07-16 12:07:38.922310231 +0000 UTC m=+37.072017649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls") pod "dns-default-f55c4" (UID: "268f88f9-d545-4d19-9bc8-8695df1dc92c") : secret "dns-default-metrics-tls" not found Jul 16 12:07:38.551634 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:38.551544 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:07:38.556004 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:38.555969 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 12:07:38.717492 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:38.717454 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r4f95" event={"ID":"94335d8a-acdd-4739-8741-ce365e66d8c5","Type":"ContainerStarted","Data":"ffd6f94a6c8cb8fe231cb3691db3b866b4239731d823036b6e60becc761728ea"} Jul 16 12:07:38.930108 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:38.929925 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:07:38.930396 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:38.930372 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:07:38.930486 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:38.930452 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls podName:268f88f9-d545-4d19-9bc8-8695df1dc92c nodeName:}" failed. No retries permitted until 2026-07-16 12:07:40.930431736 +0000 UTC m=+39.080139147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls") pod "dns-default-f55c4" (UID: "268f88f9-d545-4d19-9bc8-8695df1dc92c") : secret "dns-default-metrics-tls" not found Jul 16 12:07:40.945992 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:40.945958 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:07:40.946419 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:40.946144 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:07:40.946419 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:40.946214 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls podName:268f88f9-d545-4d19-9bc8-8695df1dc92c nodeName:}" failed. No retries permitted until 2026-07-16 12:07:44.946197969 +0000 UTC m=+43.095905380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls") pod "dns-default-f55c4" (UID: "268f88f9-d545-4d19-9bc8-8695df1dc92c") : secret "dns-default-metrics-tls" not found Jul 16 12:07:42.729259 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:42.728992 2559 generic.go:356] "Generic (PLEG): container finished" podID="cf48d5a3-b498-4329-a5ec-c0ea64596f64" containerID="b297b4f2d0539ac402478c6fc79bac5458b6f217a796ae3a01df3d1692cfa5e7" exitCode=0 Jul 16 12:07:42.729669 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:42.729066 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgwg5" event={"ID":"cf48d5a3-b498-4329-a5ec-c0ea64596f64","Type":"ContainerDied","Data":"b297b4f2d0539ac402478c6fc79bac5458b6f217a796ae3a01df3d1692cfa5e7"} Jul 16 12:07:42.730798 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:42.730772 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r4f95" event={"ID":"94335d8a-acdd-4739-8741-ce365e66d8c5","Type":"ContainerStarted","Data":"632dab9b0fbe196472fa1283cd682fc053d96537ffbdf36787cbb1ebe4210037"} Jul 16 12:07:43.735536 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:43.735497 2559 generic.go:356] "Generic (PLEG): container finished" podID="cf48d5a3-b498-4329-a5ec-c0ea64596f64" containerID="6da0061c78cb9421d022baa771cb67ca77db7d15fdb9a821d3f6ae4f3c14c66c" exitCode=0 Jul 16 12:07:43.735938 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:43.735592 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgwg5" event={"ID":"cf48d5a3-b498-4329-a5ec-c0ea64596f64","Type":"ContainerDied","Data":"6da0061c78cb9421d022baa771cb67ca77db7d15fdb9a821d3f6ae4f3c14c66c"} Jul 16 12:07:43.777596 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:43.777494 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r4f95" podStartSLOduration=2.279755337 podStartE2EDuration="6.777472628s" totalImagesPullingTime="4.497717291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:42.804980855 +0000 UTC m=+40.954688291" watchObservedRunningTime="2026-07-16 12:07:43.777472628 +0000 UTC m=+41.927180061" Jul 16 12:07:44.741722 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:44.741671 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgwg5" event={"ID":"cf48d5a3-b498-4329-a5ec-c0ea64596f64","Type":"ContainerStarted","Data":"de63fa81c98c507c460b06cc7f292bfe8f0d0728dd224d93df372fa4a1828299"} Jul 16 12:07:44.787691 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:44.787623 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pgwg5" podStartSLOduration=8.710248074 podStartE2EDuration="42.78760535s" totalImagesPullingTime="34.077357276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:07:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:44.784452424 +0000 UTC m=+42.934159858" watchObservedRunningTime="2026-07-16 12:07:44.78760535 +0000 UTC m=+42.937312782" Jul 16 12:07:44.975928 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:44.975891 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:07:44.976101 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:44.976020 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:07:44.976101 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:44.976082 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls podName:268f88f9-d545-4d19-9bc8-8695df1dc92c nodeName:}" failed. No retries permitted until 2026-07-16 12:07:52.976067978 +0000 UTC m=+51.125775389 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls") pod "dns-default-f55c4" (UID: "268f88f9-d545-4d19-9bc8-8695df1dc92c") : secret "dns-default-metrics-tls" not found Jul 16 12:07:53.030390 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:07:53.030343 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:07:53.030847 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:53.030500 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:07:53.030847 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:07:53.030567 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls podName:268f88f9-d545-4d19-9bc8-8695df1dc92c nodeName:}" failed. No retries permitted until 2026-07-16 12:08:09.03055067 +0000 UTC m=+67.180258080 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls") pod "dns-default-f55c4" (UID: "268f88f9-d545-4d19-9bc8-8695df1dc92c") : secret "dns-default-metrics-tls" not found Jul 16 12:08:03.717293 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:03.717265 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mwt4" Jul 16 12:08:08.140157 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:08.140112 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret\") pod \"global-pull-secret-syncer-mbwtf\" (UID: \"6e31220b-1279-4b66-87c2-ea2948393fd1\") " pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:08:08.140568 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:08.140164 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs\") pod \"network-metrics-daemon-2wpgm\" (UID: \"d899227b-89c3-4347-8f55-eca200398f00\") " pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:08:08.146556 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:08.146528 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 12:08:08.146698 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:08.146529 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 12:08:08.150723 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:08:08.150701 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 16 12:08:08.150786 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:08:08.150773 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs podName:d899227b-89c3-4347-8f55-eca200398f00 nodeName:}" failed. No retries permitted until 2026-07-16 12:09:12.150753059 +0000 UTC m=+130.300460473 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs") pod "network-metrics-daemon-2wpgm" (UID: "d899227b-89c3-4347-8f55-eca200398f00") : secret "metrics-daemon-secret" not found Jul 16 12:08:08.153627 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:08.153597 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e31220b-1279-4b66-87c2-ea2948393fd1-original-pull-secret\") pod \"global-pull-secret-syncer-mbwtf\" (UID: \"6e31220b-1279-4b66-87c2-ea2948393fd1\") " pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:08:08.241388 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:08.241346 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dfkw\" (UniqueName: \"kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw\") pod \"network-check-target-pgxvf\" (UID: \"ae76049f-be13-4b92-88b4-9616394c61a5\") " pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:08:08.245027 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:08.245007 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 12:08:08.259661 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:08.259637 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 12:08:08.263061 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:08.263045 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbwtf" Jul 16 12:08:08.265206 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:08.265178 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dfkw\" (UniqueName: \"kubernetes.io/projected/ae76049f-be13-4b92-88b4-9616394c61a5-kube-api-access-9dfkw\") pod \"network-check-target-pgxvf\" (UID: \"ae76049f-be13-4b92-88b4-9616394c61a5\") " pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:08:08.391975 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:08.391888 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mbwtf"] Jul 16 12:08:08.465760 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:08.465732 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hz6m9\"" Jul 16 12:08:08.473673 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:08.473649 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:08:08.622434 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:08.622391 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pgxvf"] Jul 16 12:08:08.626481 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:08:08.626451 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae76049f_be13_4b92_88b4_9616394c61a5.slice/crio-1b81097d805d6addec63305c565f06fa26ca081c20e06dc1f05c4d7b6df4ba64 WatchSource:0}: Error finding container 1b81097d805d6addec63305c565f06fa26ca081c20e06dc1f05c4d7b6df4ba64: Status 404 returned error can't find the container with id 1b81097d805d6addec63305c565f06fa26ca081c20e06dc1f05c4d7b6df4ba64 Jul 16 12:08:08.796206 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:08.796165 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pgxvf" event={"ID":"ae76049f-be13-4b92-88b4-9616394c61a5","Type":"ContainerStarted","Data":"1b81097d805d6addec63305c565f06fa26ca081c20e06dc1f05c4d7b6df4ba64"} Jul 16 12:08:08.797191 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:08.797166 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mbwtf" event={"ID":"6e31220b-1279-4b66-87c2-ea2948393fd1","Type":"ContainerStarted","Data":"2c6160809b2c9931dfeb374c994385cd86484255107c4376ca3cd2754e73ac90"} Jul 16 12:08:09.047851 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:09.047764 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:08:09.048026 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:08:09.047882 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:08:09.048026 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:08:09.047944 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls podName:268f88f9-d545-4d19-9bc8-8695df1dc92c nodeName:}" failed. No retries permitted until 2026-07-16 12:08:41.047930276 +0000 UTC m=+99.197637687 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls") pod "dns-default-f55c4" (UID: "268f88f9-d545-4d19-9bc8-8695df1dc92c") : secret "dns-default-metrics-tls" not found Jul 16 12:08:13.811654 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:13.811554 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mbwtf" event={"ID":"6e31220b-1279-4b66-87c2-ea2948393fd1","Type":"ContainerStarted","Data":"17cbc940bd001fb49c21a7c2ac528340b4f171e8f2b8107c4b8912286511009f"} Jul 16 12:08:13.813000 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:13.812965 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pgxvf" event={"ID":"ae76049f-be13-4b92-88b4-9616394c61a5","Type":"ContainerStarted","Data":"2b61e8ca922c041cebf793cd70aaeafe7a44b542ab7203f6cd2d6f08662c7684"} Jul 16 12:08:13.813122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:13.813096 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:08:13.838694 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:13.838620 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mbwtf" podStartSLOduration=66.721905204 podStartE2EDuration="1m11.83860683s" totalImagesPullingTime="5.116701626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:08:13.837507003 +0000 UTC m=+71.987214430" watchObservedRunningTime="2026-07-16 12:08:13.83860683 +0000 UTC m=+71.988314264" Jul 16 12:08:13.882875 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:13.882821 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pgxvf" podStartSLOduration=66.996950745 podStartE2EDuration="1m11.882805471s" totalImagesPullingTime="4.885854726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:08:13.88153876 +0000 UTC m=+72.031246192" watchObservedRunningTime="2026-07-16 12:08:13.882805471 +0000 UTC m=+72.032512926" Jul 16 12:08:41.062814 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:41.062755 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:08:41.063307 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:08:41.062878 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:08:41.063307 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:08:41.062935 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls podName:268f88f9-d545-4d19-9bc8-8695df1dc92c nodeName:}" failed. No retries permitted until 2026-07-16 12:09:45.062919301 +0000 UTC m=+163.212626713 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls") pod "dns-default-f55c4" (UID: "268f88f9-d545-4d19-9bc8-8695df1dc92c") : secret "dns-default-metrics-tls" not found Jul 16 12:08:44.818503 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:08:44.818468 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pgxvf" Jul 16 12:09:12.175332 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:09:12.175292 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs\") pod \"network-metrics-daemon-2wpgm\" (UID: \"d899227b-89c3-4347-8f55-eca200398f00\") " pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:09:12.175836 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:09:12.175453 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 16 12:09:12.175836 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:09:12.175539 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs podName:d899227b-89c3-4347-8f55-eca200398f00 nodeName:}" failed. No retries permitted until 2026-07-16 12:11:14.17551834 +0000 UTC m=+252.325225751 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs") pod "network-metrics-daemon-2wpgm" (UID: "d899227b-89c3-4347-8f55-eca200398f00") : secret "metrics-daemon-secret" not found Jul 16 12:09:40.286017 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:09:40.285958 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-f55c4" podUID="268f88f9-d545-4d19-9bc8-8695df1dc92c" Jul 16 12:09:40.569392 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:09:40.569300 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2wpgm" podUID="d899227b-89c3-4347-8f55-eca200398f00" Jul 16 12:09:41.021376 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:09:41.021351 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f55c4" Jul 16 12:09:45.097618 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:09:45.097561 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:09:45.098103 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:09:45.097737 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:09:45.098103 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:09:45.097801 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls podName:268f88f9-d545-4d19-9bc8-8695df1dc92c nodeName:}" failed. No retries permitted until 2026-07-16 12:11:47.097786422 +0000 UTC m=+285.247493832 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls") pod "dns-default-f55c4" (UID: "268f88f9-d545-4d19-9bc8-8695df1dc92c") : secret "dns-default-metrics-tls" not found Jul 16 12:09:55.551123 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:09:55.551019 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:10:38.667856 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.667818 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b"] Jul 16 12:10:38.670848 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.670827 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55bc8548d9-6d7s8"] Jul 16 12:10:38.670990 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.670970 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" Jul 16 12:10:38.674168 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.674148 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8"] Jul 16 12:10:38.674287 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.674223 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.674287 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.674149 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 16 12:10:38.678461 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.676311 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 12:10:38.678461 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.678157 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gtvg7\"" Jul 16 12:10:38.678461 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.678439 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 12:10:38.678461 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.678446 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 16 12:10:38.679859 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.679835 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8" Jul 16 12:10:38.680571 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.680555 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 12:10:38.681273 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.681259 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-knmbz\"" Jul 16 12:10:38.682196 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.682180 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 12:10:38.683598 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.683579 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 12:10:38.683941 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.683927 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 16 12:10:38.684322 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.684300 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:10:38.684429 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.684361 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ngs8q\"" Jul 16 12:10:38.685193 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.685172 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 16 12:10:38.694791 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.694769 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 12:10:38.695436 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.695414 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b"] Jul 16 12:10:38.696077 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.696055 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55bc8548d9-6d7s8"] Jul 16 12:10:38.706149 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.706124 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8"] Jul 16 12:10:38.736751 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.736715 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-mx7qx"] Jul 16 12:10:38.739756 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.739738 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-mx7qx" Jul 16 12:10:38.746796 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.746772 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 16 12:10:38.747057 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.747039 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-54q84\"" Jul 16 12:10:38.747148 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.747068 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 16 12:10:38.765107 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.765077 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-mx7qx"] Jul 16 12:10:38.768168 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.768143 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4lkc\" (UniqueName: \"kubernetes.io/projected/69e2cf70-9821-459a-a552-beb17a33e0e1-kube-api-access-j4lkc\") pod \"cluster-samples-operator-65db858894-ckjk8\" (UID: \"69e2cf70-9821-459a-a552-beb17a33e0e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8" Jul 16 12:10:38.768322 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.768173 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf0147eb-963a-4fde-8d30-68fa336b7244-ca-trust-extracted\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.768322 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.768193 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-certificates\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.768322 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.768220 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf0147eb-963a-4fde-8d30-68fa336b7244-trusted-ca\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.768322 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.768256 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.768322 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.768271 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7h7r\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-kube-api-access-g7h7r\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.768322 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.768287 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6djj\" (UniqueName: \"kubernetes.io/projected/b8cae4b2-a2c7-420d-832e-37982ffe2678-kube-api-access-v6djj\") pod \"cluster-monitoring-operator-5f7f494dd4-8rx2b\" (UID: \"b8cae4b2-a2c7-420d-832e-37982ffe2678\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" Jul 16 12:10:38.768322 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.768312 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-bound-sa-token\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.768621 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.768332 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-8rx2b\" (UID: \"b8cae4b2-a2c7-420d-832e-37982ffe2678\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" Jul 16 12:10:38.768621 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.768350 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf0147eb-963a-4fde-8d30-68fa336b7244-installation-pull-secrets\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.768621 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.768365 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-ckjk8\" (UID: \"69e2cf70-9821-459a-a552-beb17a33e0e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8" Jul 16 12:10:38.768621 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.768441 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b8cae4b2-a2c7-420d-832e-37982ffe2678-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-8rx2b\" (UID: \"b8cae4b2-a2c7-420d-832e-37982ffe2678\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" Jul 16 12:10:38.768621 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.768540 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf0147eb-963a-4fde-8d30-68fa336b7244-image-registry-private-configuration\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.826285 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.826251 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l"] Jul 16 12:10:38.829103 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.829086 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l" Jul 16 12:10:38.833844 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.833818 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 16 12:10:38.834057 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.833854 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:10:38.834057 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.833856 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 16 12:10:38.834203 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.834188 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qn28b\"" Jul 16 12:10:38.834277 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.834261 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 16 12:10:38.842651 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.842627 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l"] Jul 16 12:10:38.869219 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.869193 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-bound-sa-token\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.869219 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.869223 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-8rx2b\" (UID: \"b8cae4b2-a2c7-420d-832e-37982ffe2678\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" Jul 16 12:10:38.869421 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.869244 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf0147eb-963a-4fde-8d30-68fa336b7244-installation-pull-secrets\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.869421 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.869260 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-ckjk8\" (UID: \"69e2cf70-9821-459a-a552-beb17a33e0e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8" Jul 16 12:10:38.869421 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.869283 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b8cae4b2-a2c7-420d-832e-37982ffe2678-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-8rx2b\" (UID: \"b8cae4b2-a2c7-420d-832e-37982ffe2678\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" Jul 16 12:10:38.869421 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:38.869351 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 12:10:38.869421 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:38.869374 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 12:10:38.869421 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:38.869420 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls podName:b8cae4b2-a2c7-420d-832e-37982ffe2678 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:39.369401209 +0000 UTC m=+217.519108633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-8rx2b" (UID: "b8cae4b2-a2c7-420d-832e-37982ffe2678") : secret "cluster-monitoring-operator-tls" not found Jul 16 12:10:38.869737 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:38.869439 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls podName:69e2cf70-9821-459a-a552-beb17a33e0e1 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:39.369430076 +0000 UTC m=+217.519137488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls") pod "cluster-samples-operator-65db858894-ckjk8" (UID: "69e2cf70-9821-459a-a552-beb17a33e0e1") : secret "samples-operator-tls" not found Jul 16 12:10:38.869737 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.869355 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf0147eb-963a-4fde-8d30-68fa336b7244-image-registry-private-configuration\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.869737 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.869472 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4lkc\" (UniqueName: \"kubernetes.io/projected/69e2cf70-9821-459a-a552-beb17a33e0e1-kube-api-access-j4lkc\") pod \"cluster-samples-operator-65db858894-ckjk8\" (UID: \"69e2cf70-9821-459a-a552-beb17a33e0e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8" Jul 16 12:10:38.869737 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.869500 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-mx7qx\" (UID: \"3a2c5faa-25b2-4e17-852d-56aaf8403d1f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-mx7qx" Jul 16 12:10:38.869737 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.869544 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf0147eb-963a-4fde-8d30-68fa336b7244-ca-trust-extracted\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.869737 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.869569 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-certificates\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.869737 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.869600 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-nginx-conf\") pod \"networking-console-plugin-6cf856758c-mx7qx\" (UID: \"3a2c5faa-25b2-4e17-852d-56aaf8403d1f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-mx7qx" Jul 16 12:10:38.869737 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.869638 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf0147eb-963a-4fde-8d30-68fa336b7244-trusted-ca\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.869737 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.869700 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.870210 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.869750 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7h7r\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-kube-api-access-g7h7r\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.870210 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.869779 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6djj\" (UniqueName: \"kubernetes.io/projected/b8cae4b2-a2c7-420d-832e-37982ffe2678-kube-api-access-v6djj\") pod \"cluster-monitoring-operator-5f7f494dd4-8rx2b\" (UID: \"b8cae4b2-a2c7-420d-832e-37982ffe2678\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" Jul 16 12:10:38.870210 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.869948 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf0147eb-963a-4fde-8d30-68fa336b7244-ca-trust-extracted\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.870210 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:38.870060 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:10:38.870210 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:38.870074 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bc8548d9-6d7s8: secret "image-registry-tls" not found Jul 16 12:10:38.870210 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:38.870135 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls podName:bf0147eb-963a-4fde-8d30-68fa336b7244 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:39.370117849 +0000 UTC m=+217.519825275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls") pod "image-registry-55bc8548d9-6d7s8" (UID: "bf0147eb-963a-4fde-8d30-68fa336b7244") : secret "image-registry-tls" not found Jul 16 12:10:38.870210 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.870161 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b8cae4b2-a2c7-420d-832e-37982ffe2678-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-8rx2b\" (UID: \"b8cae4b2-a2c7-420d-832e-37982ffe2678\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" Jul 16 12:10:38.870210 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.870201 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-certificates\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.870659 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.870642 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf0147eb-963a-4fde-8d30-68fa336b7244-trusted-ca\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.871783 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.871761 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf0147eb-963a-4fde-8d30-68fa336b7244-installation-pull-secrets\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.871871 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.871793 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf0147eb-963a-4fde-8d30-68fa336b7244-image-registry-private-configuration\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.880259 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.880206 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-bound-sa-token\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.880397 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.880376 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6djj\" (UniqueName: \"kubernetes.io/projected/b8cae4b2-a2c7-420d-832e-37982ffe2678-kube-api-access-v6djj\") pod \"cluster-monitoring-operator-5f7f494dd4-8rx2b\" (UID: \"b8cae4b2-a2c7-420d-832e-37982ffe2678\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" Jul 16 12:10:38.880619 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.880603 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7h7r\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-kube-api-access-g7h7r\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:38.880654 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.880621 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4lkc\" (UniqueName: \"kubernetes.io/projected/69e2cf70-9821-459a-a552-beb17a33e0e1-kube-api-access-j4lkc\") pod \"cluster-samples-operator-65db858894-ckjk8\" (UID: \"69e2cf70-9821-459a-a552-beb17a33e0e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8" Jul 16 12:10:38.970362 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.970274 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hskw7\" (UniqueName: \"kubernetes.io/projected/aa1dbbae-768e-4fae-bc87-177369420431-kube-api-access-hskw7\") pod \"service-ca-operator-7695fc6c8c-vdv2l\" (UID: \"aa1dbbae-768e-4fae-bc87-177369420431\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l" Jul 16 12:10:38.970362 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.970325 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa1dbbae-768e-4fae-bc87-177369420431-serving-cert\") pod \"service-ca-operator-7695fc6c8c-vdv2l\" (UID: \"aa1dbbae-768e-4fae-bc87-177369420431\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l" Jul 16 12:10:38.970531 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.970419 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-mx7qx\" (UID: \"3a2c5faa-25b2-4e17-852d-56aaf8403d1f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-mx7qx" Jul 16 12:10:38.970531 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.970453 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-nginx-conf\") pod \"networking-console-plugin-6cf856758c-mx7qx\" (UID: \"3a2c5faa-25b2-4e17-852d-56aaf8403d1f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-mx7qx" Jul 16 12:10:38.970531 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.970472 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa1dbbae-768e-4fae-bc87-177369420431-config\") pod \"service-ca-operator-7695fc6c8c-vdv2l\" (UID: \"aa1dbbae-768e-4fae-bc87-177369420431\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l" Jul 16 12:10:38.970623 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:38.970559 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 16 12:10:38.970655 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:38.970631 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert podName:3a2c5faa-25b2-4e17-852d-56aaf8403d1f nodeName:}" failed. No retries permitted until 2026-07-16 12:10:39.470611443 +0000 UTC m=+217.620318860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-mx7qx" (UID: "3a2c5faa-25b2-4e17-852d-56aaf8403d1f") : secret "networking-console-plugin-cert" not found Jul 16 12:10:38.971095 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:38.971077 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-nginx-conf\") pod \"networking-console-plugin-6cf856758c-mx7qx\" (UID: \"3a2c5faa-25b2-4e17-852d-56aaf8403d1f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-mx7qx" Jul 16 12:10:39.071296 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:39.071253 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa1dbbae-768e-4fae-bc87-177369420431-config\") pod \"service-ca-operator-7695fc6c8c-vdv2l\" (UID: \"aa1dbbae-768e-4fae-bc87-177369420431\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l" Jul 16 12:10:39.071476 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:39.071335 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hskw7\" (UniqueName: \"kubernetes.io/projected/aa1dbbae-768e-4fae-bc87-177369420431-kube-api-access-hskw7\") pod \"service-ca-operator-7695fc6c8c-vdv2l\" (UID: \"aa1dbbae-768e-4fae-bc87-177369420431\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l" Jul 16 12:10:39.071476 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:39.071370 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa1dbbae-768e-4fae-bc87-177369420431-serving-cert\") pod \"service-ca-operator-7695fc6c8c-vdv2l\" (UID: \"aa1dbbae-768e-4fae-bc87-177369420431\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l" Jul 16 12:10:39.071902 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:39.071878 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa1dbbae-768e-4fae-bc87-177369420431-config\") pod \"service-ca-operator-7695fc6c8c-vdv2l\" (UID: \"aa1dbbae-768e-4fae-bc87-177369420431\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l" Jul 16 12:10:39.073576 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:39.073554 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa1dbbae-768e-4fae-bc87-177369420431-serving-cert\") pod \"service-ca-operator-7695fc6c8c-vdv2l\" (UID: \"aa1dbbae-768e-4fae-bc87-177369420431\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l" Jul 16 12:10:39.089460 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:39.089436 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hskw7\" (UniqueName: \"kubernetes.io/projected/aa1dbbae-768e-4fae-bc87-177369420431-kube-api-access-hskw7\") pod \"service-ca-operator-7695fc6c8c-vdv2l\" (UID: \"aa1dbbae-768e-4fae-bc87-177369420431\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l" Jul 16 12:10:39.138021 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:39.137987 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l" Jul 16 12:10:39.260248 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:39.260213 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l"] Jul 16 12:10:39.263935 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:10:39.263907 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1dbbae_768e_4fae_bc87_177369420431.slice/crio-5502570cfe7e364dd436638bbcc440978e0bc1ace3bafafad39b03fae090b872 WatchSource:0}: Error finding container 5502570cfe7e364dd436638bbcc440978e0bc1ace3bafafad39b03fae090b872: Status 404 returned error can't find the container with id 5502570cfe7e364dd436638bbcc440978e0bc1ace3bafafad39b03fae090b872 Jul 16 12:10:39.373822 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:39.373784 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:39.373822 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:39.373827 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-8rx2b\" (UID: \"b8cae4b2-a2c7-420d-832e-37982ffe2678\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" Jul 16 12:10:39.374027 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:39.373849 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-ckjk8\" (UID: \"69e2cf70-9821-459a-a552-beb17a33e0e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8" Jul 16 12:10:39.374027 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:39.373933 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 12:10:39.374027 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:39.373943 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:10:39.374027 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:39.373979 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bc8548d9-6d7s8: secret "image-registry-tls" not found Jul 16 12:10:39.374027 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:39.373992 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls podName:69e2cf70-9821-459a-a552-beb17a33e0e1 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:40.373976101 +0000 UTC m=+218.523683511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls") pod "cluster-samples-operator-65db858894-ckjk8" (UID: "69e2cf70-9821-459a-a552-beb17a33e0e1") : secret "samples-operator-tls" not found Jul 16 12:10:39.374027 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:39.374026 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls podName:bf0147eb-963a-4fde-8d30-68fa336b7244 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:40.374011921 +0000 UTC m=+218.523719336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls") pod "image-registry-55bc8548d9-6d7s8" (UID: "bf0147eb-963a-4fde-8d30-68fa336b7244") : secret "image-registry-tls" not found Jul 16 12:10:39.374251 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:39.373940 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 12:10:39.374251 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:39.374063 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls podName:b8cae4b2-a2c7-420d-832e-37982ffe2678 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:40.37405491 +0000 UTC m=+218.523762321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-8rx2b" (UID: "b8cae4b2-a2c7-420d-832e-37982ffe2678") : secret "cluster-monitoring-operator-tls" not found Jul 16 12:10:39.475252 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:39.475213 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-mx7qx\" (UID: \"3a2c5faa-25b2-4e17-852d-56aaf8403d1f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-mx7qx" Jul 16 12:10:39.475449 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:39.475399 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 16 12:10:39.475513 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:39.475481 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert podName:3a2c5faa-25b2-4e17-852d-56aaf8403d1f nodeName:}" failed. No retries permitted until 2026-07-16 12:10:40.475459934 +0000 UTC m=+218.625167366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-mx7qx" (UID: "3a2c5faa-25b2-4e17-852d-56aaf8403d1f") : secret "networking-console-plugin-cert" not found Jul 16 12:10:40.160480 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:40.160437 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l" event={"ID":"aa1dbbae-768e-4fae-bc87-177369420431","Type":"ContainerStarted","Data":"5502570cfe7e364dd436638bbcc440978e0bc1ace3bafafad39b03fae090b872"} Jul 16 12:10:40.382668 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:40.382628 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:40.382668 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:40.382673 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-8rx2b\" (UID: \"b8cae4b2-a2c7-420d-832e-37982ffe2678\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" Jul 16 12:10:40.382902 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:40.382731 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-ckjk8\" (UID: \"69e2cf70-9821-459a-a552-beb17a33e0e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8" Jul 16 12:10:40.382902 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:40.382793 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:10:40.382902 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:40.382813 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bc8548d9-6d7s8: secret "image-registry-tls" not found Jul 16 12:10:40.382902 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:40.382860 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls podName:bf0147eb-963a-4fde-8d30-68fa336b7244 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:42.382845597 +0000 UTC m=+220.532553008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls") pod "image-registry-55bc8548d9-6d7s8" (UID: "bf0147eb-963a-4fde-8d30-68fa336b7244") : secret "image-registry-tls" not found Jul 16 12:10:40.382902 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:40.382867 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 12:10:40.383164 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:40.382873 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 12:10:40.383164 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:40.382922 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls podName:b8cae4b2-a2c7-420d-832e-37982ffe2678 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:42.382910103 +0000 UTC m=+220.532617519 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-8rx2b" (UID: "b8cae4b2-a2c7-420d-832e-37982ffe2678") : secret "cluster-monitoring-operator-tls" not found Jul 16 12:10:40.383164 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:40.382982 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls podName:69e2cf70-9821-459a-a552-beb17a33e0e1 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:42.382969616 +0000 UTC m=+220.532677026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls") pod "cluster-samples-operator-65db858894-ckjk8" (UID: "69e2cf70-9821-459a-a552-beb17a33e0e1") : secret "samples-operator-tls" not found Jul 16 12:10:40.483422 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:40.483329 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-mx7qx\" (UID: \"3a2c5faa-25b2-4e17-852d-56aaf8403d1f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-mx7qx" Jul 16 12:10:40.483594 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:40.483488 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 16 12:10:40.483594 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:40.483558 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert podName:3a2c5faa-25b2-4e17-852d-56aaf8403d1f nodeName:}" failed. No retries permitted until 2026-07-16 12:10:42.483539034 +0000 UTC m=+220.633246447 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-mx7qx" (UID: "3a2c5faa-25b2-4e17-852d-56aaf8403d1f") : secret "networking-console-plugin-cert" not found Jul 16 12:10:42.166804 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:42.166763 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l" event={"ID":"aa1dbbae-768e-4fae-bc87-177369420431","Type":"ContainerStarted","Data":"77186b665106e3a1d0327be5265301aebb9fe6966d98c296c3055557584ee8f3"} Jul 16 12:10:42.188121 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:42.188070 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l" podStartSLOduration=1.908168892 podStartE2EDuration="4.188055005s" totalImagesPullingTime="2.279886113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:10:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:10:42.187725565 +0000 UTC m=+220.337432995" watchObservedRunningTime="2026-07-16 12:10:42.188055005 +0000 UTC m=+220.337762438" Jul 16 12:10:42.398112 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:42.398068 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:42.398304 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:42.398126 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-8rx2b\" (UID: \"b8cae4b2-a2c7-420d-832e-37982ffe2678\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" Jul 16 12:10:42.398304 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:42.398156 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-ckjk8\" (UID: \"69e2cf70-9821-459a-a552-beb17a33e0e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8" Jul 16 12:10:42.398304 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:42.398285 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 12:10:42.398459 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:42.398346 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls podName:69e2cf70-9821-459a-a552-beb17a33e0e1 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:46.398328615 +0000 UTC m=+224.548036026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls") pod "cluster-samples-operator-65db858894-ckjk8" (UID: "69e2cf70-9821-459a-a552-beb17a33e0e1") : secret "samples-operator-tls" not found Jul 16 12:10:42.398810 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:42.398786 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 12:10:42.398903 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:42.398851 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls podName:b8cae4b2-a2c7-420d-832e-37982ffe2678 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:46.398833081 +0000 UTC m=+224.548540499 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-8rx2b" (UID: "b8cae4b2-a2c7-420d-832e-37982ffe2678") : secret "cluster-monitoring-operator-tls" not found Jul 16 12:10:42.398962 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:42.398935 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:10:42.398962 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:42.398947 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bc8548d9-6d7s8: secret "image-registry-tls" not found Jul 16 12:10:42.399057 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:42.398979 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls podName:bf0147eb-963a-4fde-8d30-68fa336b7244 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:46.398968302 +0000 UTC m=+224.548675718 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls") pod "image-registry-55bc8548d9-6d7s8" (UID: "bf0147eb-963a-4fde-8d30-68fa336b7244") : secret "image-registry-tls" not found Jul 16 12:10:42.498894 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:42.498837 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-mx7qx\" (UID: \"3a2c5faa-25b2-4e17-852d-56aaf8403d1f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-mx7qx" Jul 16 12:10:42.499073 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:42.498988 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 16 12:10:42.499073 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:42.499053 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert podName:3a2c5faa-25b2-4e17-852d-56aaf8403d1f nodeName:}" failed. No retries permitted until 2026-07-16 12:10:46.499037788 +0000 UTC m=+224.648745202 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-mx7qx" (UID: "3a2c5faa-25b2-4e17-852d-56aaf8403d1f") : secret "networking-console-plugin-cert" not found Jul 16 12:10:43.762728 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:43.762691 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qlcvh"] Jul 16 12:10:43.766094 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:43.766070 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:43.773963 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:43.773932 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 12:10:43.774065 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:43.773974 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8fngt\"" Jul 16 12:10:43.774768 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:43.774755 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 12:10:43.774811 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:43.774794 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 12:10:43.775008 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:43.774990 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 12:10:43.788949 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:43.788922 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qlcvh"] Jul 16 12:10:43.911441 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:43.911395 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:43.911441 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:43.911441 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jztvq\" (UniqueName: \"kubernetes.io/projected/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-api-access-jztvq\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:43.911649 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:43.911525 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8a0c36a7-ae21-445b-863b-bab16f7d69b7-crio-socket\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:43.911649 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:43.911563 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a0c36a7-ae21-445b-863b-bab16f7d69b7-data-volume\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:43.911649 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:43.911593 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:44.012273 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.012243 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:44.012348 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.012284 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jztvq\" (UniqueName: \"kubernetes.io/projected/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-api-access-jztvq\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:44.012388 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:44.012358 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:10:44.012427 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.012405 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8a0c36a7-ae21-445b-863b-bab16f7d69b7-crio-socket\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:44.012466 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:44.012425 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm podName:8a0c36a7-ae21-445b-863b-bab16f7d69b7 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:44.512408148 +0000 UTC m=+222.662115599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qlcvh" (UID: "8a0c36a7-ae21-445b-863b-bab16f7d69b7") : configmap "kube-rbac-proxy" not found Jul 16 12:10:44.012521 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.012459 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8a0c36a7-ae21-445b-863b-bab16f7d69b7-crio-socket\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:44.012521 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.012452 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a0c36a7-ae21-445b-863b-bab16f7d69b7-data-volume\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:44.012639 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.012522 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:44.012722 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:44.012653 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:10:44.012794 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:44.012729 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls podName:8a0c36a7-ae21-445b-863b-bab16f7d69b7 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:44.512712692 +0000 UTC m=+222.662420113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qlcvh" (UID: "8a0c36a7-ae21-445b-863b-bab16f7d69b7") : secret "insights-runtime-extractor-tls" not found Jul 16 12:10:44.012794 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.012750 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a0c36a7-ae21-445b-863b-bab16f7d69b7-data-volume\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:44.023766 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.023736 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jztvq\" (UniqueName: \"kubernetes.io/projected/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-api-access-jztvq\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:44.516318 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.516280 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:44.516551 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:44.516432 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:10:44.516551 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:44.516487 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:10:44.516551 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:44.516497 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls podName:8a0c36a7-ae21-445b-863b-bab16f7d69b7 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:45.516480605 +0000 UTC m=+223.666188015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qlcvh" (UID: "8a0c36a7-ae21-445b-863b-bab16f7d69b7") : secret "insights-runtime-extractor-tls" not found Jul 16 12:10:44.516551 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.516442 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:44.516551 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:44.516534 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm podName:8a0c36a7-ae21-445b-863b-bab16f7d69b7 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:45.516517812 +0000 UTC m=+223.666225223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qlcvh" (UID: "8a0c36a7-ae21-445b-863b-bab16f7d69b7") : configmap "kube-rbac-proxy" not found Jul 16 12:10:44.749061 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.749021 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-pdf7g"] Jul 16 12:10:44.752244 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.752221 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-pdf7g" Jul 16 12:10:44.755161 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.755135 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 16 12:10:44.757033 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.757015 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 16 12:10:44.757136 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.757034 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 16 12:10:44.757296 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.757282 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-472tb\"" Jul 16 12:10:44.758144 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.758127 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 16 12:10:44.763344 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.763323 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-pdf7g"] Jul 16 12:10:44.920526 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.920432 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pdcv\" (UniqueName: \"kubernetes.io/projected/48db7d86-bec5-4adf-ba87-c450d9ec0b2a-kube-api-access-9pdcv\") pod \"service-ca-79f68b9f4f-pdf7g\" (UID: \"48db7d86-bec5-4adf-ba87-c450d9ec0b2a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-pdf7g" Jul 16 12:10:44.920526 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.920474 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/48db7d86-bec5-4adf-ba87-c450d9ec0b2a-signing-key\") pod \"service-ca-79f68b9f4f-pdf7g\" (UID: \"48db7d86-bec5-4adf-ba87-c450d9ec0b2a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-pdf7g" Jul 16 12:10:44.920791 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:44.920596 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/48db7d86-bec5-4adf-ba87-c450d9ec0b2a-signing-cabundle\") pod \"service-ca-79f68b9f4f-pdf7g\" (UID: \"48db7d86-bec5-4adf-ba87-c450d9ec0b2a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-pdf7g" Jul 16 12:10:45.022005 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:45.021966 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdcv\" (UniqueName: \"kubernetes.io/projected/48db7d86-bec5-4adf-ba87-c450d9ec0b2a-kube-api-access-9pdcv\") pod \"service-ca-79f68b9f4f-pdf7g\" (UID: \"48db7d86-bec5-4adf-ba87-c450d9ec0b2a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-pdf7g" Jul 16 12:10:45.022005 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:45.022004 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/48db7d86-bec5-4adf-ba87-c450d9ec0b2a-signing-key\") pod \"service-ca-79f68b9f4f-pdf7g\" (UID: \"48db7d86-bec5-4adf-ba87-c450d9ec0b2a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-pdf7g" Jul 16 12:10:45.022213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:45.022191 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/48db7d86-bec5-4adf-ba87-c450d9ec0b2a-signing-cabundle\") pod \"service-ca-79f68b9f4f-pdf7g\" (UID: \"48db7d86-bec5-4adf-ba87-c450d9ec0b2a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-pdf7g" Jul 16 12:10:45.022830 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:45.022804 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/48db7d86-bec5-4adf-ba87-c450d9ec0b2a-signing-cabundle\") pod \"service-ca-79f68b9f4f-pdf7g\" (UID: \"48db7d86-bec5-4adf-ba87-c450d9ec0b2a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-pdf7g" Jul 16 12:10:45.024485 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:45.024463 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/48db7d86-bec5-4adf-ba87-c450d9ec0b2a-signing-key\") pod \"service-ca-79f68b9f4f-pdf7g\" (UID: \"48db7d86-bec5-4adf-ba87-c450d9ec0b2a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-pdf7g" Jul 16 12:10:45.032656 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:45.032634 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pdcv\" (UniqueName: \"kubernetes.io/projected/48db7d86-bec5-4adf-ba87-c450d9ec0b2a-kube-api-access-9pdcv\") pod \"service-ca-79f68b9f4f-pdf7g\" (UID: \"48db7d86-bec5-4adf-ba87-c450d9ec0b2a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-pdf7g" Jul 16 12:10:45.061286 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:45.061243 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-pdf7g" Jul 16 12:10:45.204903 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:45.204820 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-pdf7g"] Jul 16 12:10:45.207985 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:10:45.207957 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48db7d86_bec5_4adf_ba87_c450d9ec0b2a.slice/crio-bfecb023b57afebed99fdc602bc434536d90c96cb97db9fdfbcff5baf365bd29 WatchSource:0}: Error finding container bfecb023b57afebed99fdc602bc434536d90c96cb97db9fdfbcff5baf365bd29: Status 404 returned error can't find the container with id bfecb023b57afebed99fdc602bc434536d90c96cb97db9fdfbcff5baf365bd29 Jul 16 12:10:45.526172 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:45.526133 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:45.526365 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:45.526231 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:45.526365 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:45.526273 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:10:45.526365 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:45.526329 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:10:45.526365 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:45.526353 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls podName:8a0c36a7-ae21-445b-863b-bab16f7d69b7 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:47.526337276 +0000 UTC m=+225.676044687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qlcvh" (UID: "8a0c36a7-ae21-445b-863b-bab16f7d69b7") : secret "insights-runtime-extractor-tls" not found Jul 16 12:10:45.526516 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:45.526375 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm podName:8a0c36a7-ae21-445b-863b-bab16f7d69b7 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:47.526362861 +0000 UTC m=+225.676070271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qlcvh" (UID: "8a0c36a7-ae21-445b-863b-bab16f7d69b7") : configmap "kube-rbac-proxy" not found Jul 16 12:10:45.610312 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:45.610281 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6vr5g_d1fe16f9-e90b-4dbf-8462-92f01545a15e/dns-node-resolver/0.log" Jul 16 12:10:46.180284 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:46.180245 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-pdf7g" event={"ID":"48db7d86-bec5-4adf-ba87-c450d9ec0b2a","Type":"ContainerStarted","Data":"970a9c8ff67f95140069ffa846321730746593041c8133706db9198f7129d4cd"} Jul 16 12:10:46.180712 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:46.180303 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-pdf7g" event={"ID":"48db7d86-bec5-4adf-ba87-c450d9ec0b2a","Type":"ContainerStarted","Data":"bfecb023b57afebed99fdc602bc434536d90c96cb97db9fdfbcff5baf365bd29"} Jul 16 12:10:46.203330 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:46.203269 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-pdf7g" podStartSLOduration=2.203244674 podStartE2EDuration="2.203244674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:10:46.202155365 +0000 UTC m=+224.351862799" watchObservedRunningTime="2026-07-16 12:10:46.203244674 +0000 UTC m=+224.352952110" Jul 16 12:10:46.431948 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:46.431849 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:46.431948 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:46.431901 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-8rx2b\" (UID: \"b8cae4b2-a2c7-420d-832e-37982ffe2678\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" Jul 16 12:10:46.431948 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:46.431943 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-ckjk8\" (UID: \"69e2cf70-9821-459a-a552-beb17a33e0e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8" Jul 16 12:10:46.432224 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:46.432011 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:10:46.432224 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:46.432035 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bc8548d9-6d7s8: secret "image-registry-tls" not found Jul 16 12:10:46.432224 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:46.432071 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 12:10:46.432224 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:46.432095 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls podName:bf0147eb-963a-4fde-8d30-68fa336b7244 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:54.432075803 +0000 UTC m=+232.581783217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls") pod "image-registry-55bc8548d9-6d7s8" (UID: "bf0147eb-963a-4fde-8d30-68fa336b7244") : secret "image-registry-tls" not found Jul 16 12:10:46.432224 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:46.432122 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls podName:69e2cf70-9821-459a-a552-beb17a33e0e1 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:54.432106884 +0000 UTC m=+232.581814301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls") pod "cluster-samples-operator-65db858894-ckjk8" (UID: "69e2cf70-9821-459a-a552-beb17a33e0e1") : secret "samples-operator-tls" not found Jul 16 12:10:46.432224 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:46.432176 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 12:10:46.432224 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:46.432205 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls podName:b8cae4b2-a2c7-420d-832e-37982ffe2678 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:54.43219564 +0000 UTC m=+232.581903057 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-8rx2b" (UID: "b8cae4b2-a2c7-420d-832e-37982ffe2678") : secret "cluster-monitoring-operator-tls" not found Jul 16 12:10:46.532235 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:46.532201 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-mx7qx\" (UID: \"3a2c5faa-25b2-4e17-852d-56aaf8403d1f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-mx7qx" Jul 16 12:10:46.532431 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:46.532358 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 16 12:10:46.532488 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:46.532464 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert podName:3a2c5faa-25b2-4e17-852d-56aaf8403d1f nodeName:}" failed. No retries permitted until 2026-07-16 12:10:54.532441528 +0000 UTC m=+232.682148943 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-mx7qx" (UID: "3a2c5faa-25b2-4e17-852d-56aaf8403d1f") : secret "networking-console-plugin-cert" not found Jul 16 12:10:46.555529 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:46.555504 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6wxl5_d4b17153-bc06-4588-a2ba-6a11d0de2da9/node-ca/0.log" Jul 16 12:10:47.541101 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:47.541063 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:47.541596 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:47.541167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:47.541596 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:47.541239 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:10:47.541596 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:47.541261 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:10:47.541596 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:47.541311 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm podName:8a0c36a7-ae21-445b-863b-bab16f7d69b7 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:51.541297422 +0000 UTC m=+229.691004833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qlcvh" (UID: "8a0c36a7-ae21-445b-863b-bab16f7d69b7") : configmap "kube-rbac-proxy" not found Jul 16 12:10:47.541596 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:47.541324 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls podName:8a0c36a7-ae21-445b-863b-bab16f7d69b7 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:51.541318944 +0000 UTC m=+229.691026355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qlcvh" (UID: "8a0c36a7-ae21-445b-863b-bab16f7d69b7") : secret "insights-runtime-extractor-tls" not found Jul 16 12:10:47.746935 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:47.746906 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r4f95_94335d8a-acdd-4739-8741-ce365e66d8c5/serve-healthcheck-canary/0.log" Jul 16 12:10:51.574658 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:51.574622 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:51.575044 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:51.574782 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:10:51.575044 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:51.574821 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:51.575044 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:51.574844 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls podName:8a0c36a7-ae21-445b-863b-bab16f7d69b7 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:59.574828371 +0000 UTC m=+237.724535812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qlcvh" (UID: "8a0c36a7-ae21-445b-863b-bab16f7d69b7") : secret "insights-runtime-extractor-tls" not found Jul 16 12:10:51.575044 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:51.574884 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:10:51.575044 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:10:51.574926 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm podName:8a0c36a7-ae21-445b-863b-bab16f7d69b7 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:59.574914494 +0000 UTC m=+237.724621904 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qlcvh" (UID: "8a0c36a7-ae21-445b-863b-bab16f7d69b7") : configmap "kube-rbac-proxy" not found Jul 16 12:10:54.500493 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:54.500452 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:54.500493 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:54.500497 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-8rx2b\" (UID: \"b8cae4b2-a2c7-420d-832e-37982ffe2678\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" Jul 16 12:10:54.501030 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:54.500519 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-ckjk8\" (UID: \"69e2cf70-9821-459a-a552-beb17a33e0e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8" Jul 16 12:10:54.502973 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:54.502947 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/69e2cf70-9821-459a-a552-beb17a33e0e1-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-ckjk8\" (UID: \"69e2cf70-9821-459a-a552-beb17a33e0e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8" Jul 16 12:10:54.503064 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:54.502997 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8cae4b2-a2c7-420d-832e-37982ffe2678-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-8rx2b\" (UID: \"b8cae4b2-a2c7-420d-832e-37982ffe2678\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" Jul 16 12:10:54.503064 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:54.503039 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls\") pod \"image-registry-55bc8548d9-6d7s8\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:54.583380 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:54.583337 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" Jul 16 12:10:54.589360 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:54.589333 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:54.594205 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:54.594184 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8" Jul 16 12:10:54.601031 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:54.601003 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-mx7qx\" (UID: \"3a2c5faa-25b2-4e17-852d-56aaf8403d1f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-mx7qx" Jul 16 12:10:54.603789 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:54.603767 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a2c5faa-25b2-4e17-852d-56aaf8403d1f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-mx7qx\" (UID: \"3a2c5faa-25b2-4e17-852d-56aaf8403d1f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-mx7qx" Jul 16 12:10:54.648945 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:54.648659 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-mx7qx" Jul 16 12:10:54.754045 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:54.753963 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b"] Jul 16 12:10:54.757715 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:10:54.757667 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cae4b2_a2c7_420d_832e_37982ffe2678.slice/crio-d25fb3b225b6da8e31a0b500eacc15cf58f771758f6809a1c4f074d955d01b99 WatchSource:0}: Error finding container d25fb3b225b6da8e31a0b500eacc15cf58f771758f6809a1c4f074d955d01b99: Status 404 returned error can't find the container with id d25fb3b225b6da8e31a0b500eacc15cf58f771758f6809a1c4f074d955d01b99 Jul 16 12:10:54.829775 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:54.829747 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-mx7qx"] Jul 16 12:10:54.832361 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:10:54.832337 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2c5faa_25b2_4e17_852d_56aaf8403d1f.slice/crio-5d1451a9b508aed0a0851242f88b4b3576f72f69b696a9628fe2409c0ace848a WatchSource:0}: Error finding container 5d1451a9b508aed0a0851242f88b4b3576f72f69b696a9628fe2409c0ace848a: Status 404 returned error can't find the container with id 5d1451a9b508aed0a0851242f88b4b3576f72f69b696a9628fe2409c0ace848a Jul 16 12:10:54.988903 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:54.988869 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8"] Jul 16 12:10:54.994278 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:10:54.994224 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0147eb_963a_4fde_8d30_68fa336b7244.slice/crio-bf12c5df30571bdacbabf424620f5263c4acd9b3715b39f8a57ca07530ade1d4 WatchSource:0}: Error finding container bf12c5df30571bdacbabf424620f5263c4acd9b3715b39f8a57ca07530ade1d4: Status 404 returned error can't find the container with id bf12c5df30571bdacbabf424620f5263c4acd9b3715b39f8a57ca07530ade1d4 Jul 16 12:10:54.995480 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:54.995459 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55bc8548d9-6d7s8"] Jul 16 12:10:55.207890 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:55.207849 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-mx7qx" event={"ID":"3a2c5faa-25b2-4e17-852d-56aaf8403d1f","Type":"ContainerStarted","Data":"5d1451a9b508aed0a0851242f88b4b3576f72f69b696a9628fe2409c0ace848a"} Jul 16 12:10:55.209552 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:55.209524 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" event={"ID":"bf0147eb-963a-4fde-8d30-68fa336b7244","Type":"ContainerStarted","Data":"114b245114c8a16d424d2e8593d0dee20eb89978dc0fbc24b8b7eaa1f4885b9c"} Jul 16 12:10:55.209712 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:55.209579 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" event={"ID":"bf0147eb-963a-4fde-8d30-68fa336b7244","Type":"ContainerStarted","Data":"bf12c5df30571bdacbabf424620f5263c4acd9b3715b39f8a57ca07530ade1d4"} Jul 16 12:10:55.209712 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:55.209608 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:10:55.210734 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:55.210706 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8" event={"ID":"69e2cf70-9821-459a-a552-beb17a33e0e1","Type":"ContainerStarted","Data":"6a968f2993e22eae201f29cf8c7497a96d16c95e826e1ed4abb1d20bee4f1e80"} Jul 16 12:10:55.211798 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:55.211769 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" event={"ID":"b8cae4b2-a2c7-420d-832e-37982ffe2678","Type":"ContainerStarted","Data":"d25fb3b225b6da8e31a0b500eacc15cf58f771758f6809a1c4f074d955d01b99"} Jul 16 12:10:55.234570 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:55.234517 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" podStartSLOduration=17.234498224 podStartE2EDuration="17.234498224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:10:55.234114285 +0000 UTC m=+233.383821718" watchObservedRunningTime="2026-07-16 12:10:55.234498224 +0000 UTC m=+233.384205649" Jul 16 12:10:57.222594 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:57.222501 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" event={"ID":"b8cae4b2-a2c7-420d-832e-37982ffe2678","Type":"ContainerStarted","Data":"776241bb93c781ef149f5594a0b6556e454509573e63a2ce61cd288fae7b40a7"} Jul 16 12:10:57.224041 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:57.224010 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-mx7qx" event={"ID":"3a2c5faa-25b2-4e17-852d-56aaf8403d1f","Type":"ContainerStarted","Data":"a61bbd7b674908ba64f0794b2ae53adca4fb5611f5bdc4da68ec27e49572839c"} Jul 16 12:10:57.265251 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:57.265197 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-8rx2b" podStartSLOduration=17.20156201 podStartE2EDuration="19.265182922s" totalImagesPullingTime="2.063620912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:10:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:10:57.263935684 +0000 UTC m=+235.413643117" watchObservedRunningTime="2026-07-16 12:10:57.265182922 +0000 UTC m=+235.414890355" Jul 16 12:10:57.315189 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:57.315132 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-mx7qx" podStartSLOduration=18.000799323 podStartE2EDuration="19.315113726s" totalImagesPullingTime="1.314314403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:10:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:10:57.312517566 +0000 UTC m=+235.462225008" watchObservedRunningTime="2026-07-16 12:10:57.315113726 +0000 UTC m=+235.464821162" Jul 16 12:10:58.231658 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:58.231623 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8" event={"ID":"69e2cf70-9821-459a-a552-beb17a33e0e1","Type":"ContainerStarted","Data":"08bdaf5339971af72541e7164eaaecccf039545436e0940de7966725a237c7ce"} Jul 16 12:10:58.232051 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:58.231692 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8" event={"ID":"69e2cf70-9821-459a-a552-beb17a33e0e1","Type":"ContainerStarted","Data":"0fd1f3bf2f2ca2cd3ea451a18136a0764078b2dbaf6a5ea0a2d692257ebc45b0"} Jul 16 12:10:58.260849 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:58.260798 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-ckjk8" podStartSLOduration=17.631724228 podStartE2EDuration="20.260784011s" totalImagesPullingTime="2.629059783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:10:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:10:58.254055059 +0000 UTC m=+236.403762493" watchObservedRunningTime="2026-07-16 12:10:58.260784011 +0000 UTC m=+236.410491422" Jul 16 12:10:59.644157 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:59.644110 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:59.644644 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:59.644193 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:59.644856 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:59.644833 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8a0c36a7-ae21-445b-863b-bab16f7d69b7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:59.646358 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:59.646338 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8a0c36a7-ae21-445b-863b-bab16f7d69b7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlcvh\" (UID: \"8a0c36a7-ae21-445b-863b-bab16f7d69b7\") " pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:59.674523 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:59.674495 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qlcvh" Jul 16 12:10:59.817705 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:10:59.817655 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qlcvh"] Jul 16 12:10:59.821482 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:10:59.821458 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0c36a7_ae21_445b_863b_bab16f7d69b7.slice/crio-f42aef8ec71e93a507c140a3ed94940fa1e09e8c270ef841903bd9e9cc799d23 WatchSource:0}: Error finding container f42aef8ec71e93a507c140a3ed94940fa1e09e8c270ef841903bd9e9cc799d23: Status 404 returned error can't find the container with id f42aef8ec71e93a507c140a3ed94940fa1e09e8c270ef841903bd9e9cc799d23 Jul 16 12:11:00.238891 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:00.238853 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qlcvh" event={"ID":"8a0c36a7-ae21-445b-863b-bab16f7d69b7","Type":"ContainerStarted","Data":"f42aef8ec71e93a507c140a3ed94940fa1e09e8c270ef841903bd9e9cc799d23"} Jul 16 12:11:05.259066 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:05.259023 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qlcvh" event={"ID":"8a0c36a7-ae21-445b-863b-bab16f7d69b7","Type":"ContainerStarted","Data":"90d3e2be54401ca0a68c89a890dab60133ebbc1da9fc8f4853478bf77cabf66c"} Jul 16 12:11:07.973886 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:07.973852 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4mmts"] Jul 16 12:11:07.986295 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:07.986272 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4mmts" Jul 16 12:11:08.003574 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:08.003551 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 16 12:11:08.006987 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:08.006972 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 16 12:11:08.033281 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:08.033251 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-lwbs9\"" Jul 16 12:11:08.059383 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:08.059304 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4mmts"] Jul 16 12:11:08.115699 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:08.115652 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjw8p\" (UniqueName: \"kubernetes.io/projected/8760bbb9-7c9f-4e6d-a732-5afbab2e82e9-kube-api-access-fjw8p\") pod \"migrator-7f59b54f6c-4mmts\" (UID: \"8760bbb9-7c9f-4e6d-a732-5afbab2e82e9\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4mmts" Jul 16 12:11:08.216847 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:08.216814 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjw8p\" (UniqueName: \"kubernetes.io/projected/8760bbb9-7c9f-4e6d-a732-5afbab2e82e9-kube-api-access-fjw8p\") pod \"migrator-7f59b54f6c-4mmts\" (UID: \"8760bbb9-7c9f-4e6d-a732-5afbab2e82e9\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4mmts" Jul 16 12:11:08.228618 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:08.228533 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55bc8548d9-6d7s8"] Jul 16 12:11:08.266255 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:08.266229 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjw8p\" (UniqueName: \"kubernetes.io/projected/8760bbb9-7c9f-4e6d-a732-5afbab2e82e9-kube-api-access-fjw8p\") pod \"migrator-7f59b54f6c-4mmts\" (UID: \"8760bbb9-7c9f-4e6d-a732-5afbab2e82e9\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4mmts" Jul 16 12:11:08.275827 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:08.275799 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qlcvh" event={"ID":"8a0c36a7-ae21-445b-863b-bab16f7d69b7","Type":"ContainerStarted","Data":"8cda5d2be9df3b36384c8d145af71d72e8aa48c940be6e1cd9b87169bee5f7e2"} Jul 16 12:11:08.294557 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:08.294533 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4mmts" Jul 16 12:11:08.486590 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:08.486497 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4mmts"] Jul 16 12:11:08.490396 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:11:08.490367 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8760bbb9_7c9f_4e6d_a732_5afbab2e82e9.slice/crio-41ad068ed5d2350432daae44899c193cb9f81c5c71ced35cd14f926f93d7ba4f WatchSource:0}: Error finding container 41ad068ed5d2350432daae44899c193cb9f81c5c71ced35cd14f926f93d7ba4f: Status 404 returned error can't find the container with id 41ad068ed5d2350432daae44899c193cb9f81c5c71ced35cd14f926f93d7ba4f Jul 16 12:11:09.282094 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:09.282051 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4mmts" event={"ID":"8760bbb9-7c9f-4e6d-a732-5afbab2e82e9","Type":"ContainerStarted","Data":"41ad068ed5d2350432daae44899c193cb9f81c5c71ced35cd14f926f93d7ba4f"} Jul 16 12:11:12.295018 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:12.294977 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4mmts" event={"ID":"8760bbb9-7c9f-4e6d-a732-5afbab2e82e9","Type":"ContainerStarted","Data":"9fa9aa5733b3667920a20f2900c26fc303e7a7b2e2a83e82cedb72c2dd5b6f52"} Jul 16 12:11:12.295443 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:12.295038 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4mmts" event={"ID":"8760bbb9-7c9f-4e6d-a732-5afbab2e82e9","Type":"ContainerStarted","Data":"4cc0e34ed0e6206ae68412827d02bb24afaaa281b72298eaf138cdbc6d9e520d"} Jul 16 12:11:12.296910 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:12.296888 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qlcvh" event={"ID":"8a0c36a7-ae21-445b-863b-bab16f7d69b7","Type":"ContainerStarted","Data":"d39fec3d85ecd649798fb848882c6211090c365ca6fbdd464231078cc10980b3"} Jul 16 12:11:12.319752 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:12.319672 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4mmts" podStartSLOduration=2.18026471 podStartE2EDuration="5.319659138s" totalImagesPullingTime="3.139394428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:11:12.319503495 +0000 UTC m=+250.469210929" watchObservedRunningTime="2026-07-16 12:11:12.319659138 +0000 UTC m=+250.469366568" Jul 16 12:11:12.350270 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:12.350221 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qlcvh" podStartSLOduration=17.639807264 podStartE2EDuration="29.350206212s" totalImagesPullingTime="11.710398948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:10:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:11:12.349555909 +0000 UTC m=+250.499263342" watchObservedRunningTime="2026-07-16 12:11:12.350206212 +0000 UTC m=+250.499913644" Jul 16 12:11:14.266404 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:14.266367 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs\") pod \"network-metrics-daemon-2wpgm\" (UID: \"d899227b-89c3-4347-8f55-eca200398f00\") " pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:11:14.268663 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:14.268629 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d899227b-89c3-4347-8f55-eca200398f00-metrics-certs\") pod \"network-metrics-daemon-2wpgm\" (UID: \"d899227b-89c3-4347-8f55-eca200398f00\") " pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:11:14.456402 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:14.456366 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w29rm\"" Jul 16 12:11:14.462042 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:14.462021 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wpgm" Jul 16 12:11:14.616852 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:14.616817 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2wpgm"] Jul 16 12:11:14.619717 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:11:14.619688 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd899227b_89c3_4347_8f55_eca200398f00.slice/crio-670c97f16a9dd43ee0339fc1f509bef35d35d9e41ca643700cc61014eb20c843 WatchSource:0}: Error finding container 670c97f16a9dd43ee0339fc1f509bef35d35d9e41ca643700cc61014eb20c843: Status 404 returned error can't find the container with id 670c97f16a9dd43ee0339fc1f509bef35d35d9e41ca643700cc61014eb20c843 Jul 16 12:11:15.311217 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:15.311181 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2wpgm" event={"ID":"d899227b-89c3-4347-8f55-eca200398f00","Type":"ContainerStarted","Data":"670c97f16a9dd43ee0339fc1f509bef35d35d9e41ca643700cc61014eb20c843"} Jul 16 12:11:16.316800 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.316752 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2wpgm" event={"ID":"d899227b-89c3-4347-8f55-eca200398f00","Type":"ContainerStarted","Data":"49de6e1166301ef3fd6367eb41249dd24358230335bbd07b284a13883d5be8ff"} Jul 16 12:11:16.317297 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.316819 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2wpgm" event={"ID":"d899227b-89c3-4347-8f55-eca200398f00","Type":"ContainerStarted","Data":"f2de5089c6b2c1d3ec09c17912416be12bf7cead031c952de0d1fb98c27fc250"} Jul 16 12:11:16.371843 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.371780 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2wpgm" podStartSLOduration=253.280797805 podStartE2EDuration="4m14.371764216s" totalImagesPullingTime="1.090966411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:11:16.371281931 +0000 UTC m=+254.520989365" watchObservedRunningTime="2026-07-16 12:11:16.371764216 +0000 UTC m=+254.521471692" Jul 16 12:11:16.414483 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.414445 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-f2hjc"] Jul 16 12:11:16.417686 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.417658 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.422140 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.422120 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 16 12:11:16.422621 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.422605 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 16 12:11:16.423865 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.423840 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 16 12:11:16.423999 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.423844 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 12:11:16.423999 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.423889 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-sjbs4\"" Jul 16 12:11:16.451313 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.451283 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-f2hjc"] Jul 16 12:11:16.484254 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.484214 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.484405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.484290 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.484405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.484319 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.484405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.484364 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.484405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.484390 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw96t\" (UniqueName: \"kubernetes.io/projected/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-kube-api-access-tw96t\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.484556 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.484432 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.584966 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.584877 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.584966 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.584920 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.585150 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.585072 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.585150 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.585108 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.585225 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.585164 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.585225 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.585194 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw96t\" (UniqueName: \"kubernetes.io/projected/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-kube-api-access-tw96t\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.585642 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.585617 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.585797 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.585749 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.585966 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.585942 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.587481 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.587462 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.587591 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.587574 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.621789 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.621752 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw96t\" (UniqueName: \"kubernetes.io/projected/9ac5634c-d4f8-4e17-a719-6a6dafff2d5a-kube-api-access-tw96t\") pod \"kube-state-metrics-785778ff74-f2hjc\" (UID: \"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.677552 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.677517 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g"] Jul 16 12:11:16.681737 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.681720 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" Jul 16 12:11:16.686106 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.686085 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 16 12:11:16.686536 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.686519 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6h5sv\"" Jul 16 12:11:16.687129 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.687108 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 16 12:11:16.728460 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.728431 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" Jul 16 12:11:16.731868 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.731848 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c2r95"] Jul 16 12:11:16.736212 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.736188 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.739765 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.739730 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g"] Jul 16 12:11:16.740991 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.740969 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 12:11:16.741637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.741616 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 12:11:16.744366 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.742101 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wcl2h\"" Jul 16 12:11:16.747740 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.747189 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 12:11:16.787370 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.787330 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/231fbb85-1b53-4342-99bc-a25f64972033-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-wsr5g\" (UID: \"231fbb85-1b53-4342-99bc-a25f64972033\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" Jul 16 12:11:16.787622 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.787599 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/231fbb85-1b53-4342-99bc-a25f64972033-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-wsr5g\" (UID: \"231fbb85-1b53-4342-99bc-a25f64972033\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" Jul 16 12:11:16.787854 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.787832 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/231fbb85-1b53-4342-99bc-a25f64972033-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-wsr5g\" (UID: \"231fbb85-1b53-4342-99bc-a25f64972033\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" Jul 16 12:11:16.788006 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.787972 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlh79\" (UniqueName: \"kubernetes.io/projected/231fbb85-1b53-4342-99bc-a25f64972033-kube-api-access-zlh79\") pod \"openshift-state-metrics-84bd68b896-wsr5g\" (UID: \"231fbb85-1b53-4342-99bc-a25f64972033\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" Jul 16 12:11:16.888419 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.888326 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-f2hjc"] Jul 16 12:11:16.888611 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.888586 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42b9dc92-8c58-474e-aa83-806dee5ed2df-root\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.888707 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.888635 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42b9dc92-8c58-474e-aa83-806dee5ed2df-node-exporter-textfile\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.888707 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.888688 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42b9dc92-8c58-474e-aa83-806dee5ed2df-node-exporter-accelerators-collector-config\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.888838 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.888729 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/231fbb85-1b53-4342-99bc-a25f64972033-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-wsr5g\" (UID: \"231fbb85-1b53-4342-99bc-a25f64972033\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" Jul 16 12:11:16.888838 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.888748 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/231fbb85-1b53-4342-99bc-a25f64972033-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-wsr5g\" (UID: \"231fbb85-1b53-4342-99bc-a25f64972033\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" Jul 16 12:11:16.888838 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.888768 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42b9dc92-8c58-474e-aa83-806dee5ed2df-node-exporter-wtmp\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.888838 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.888792 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42b9dc92-8c58-474e-aa83-806dee5ed2df-sys\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.888838 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.888809 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42b9dc92-8c58-474e-aa83-806dee5ed2df-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.889140 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.888862 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/231fbb85-1b53-4342-99bc-a25f64972033-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-wsr5g\" (UID: \"231fbb85-1b53-4342-99bc-a25f64972033\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" Jul 16 12:11:16.889140 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.888894 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42b9dc92-8c58-474e-aa83-806dee5ed2df-node-exporter-tls\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.889140 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.888924 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlh79\" (UniqueName: \"kubernetes.io/projected/231fbb85-1b53-4342-99bc-a25f64972033-kube-api-access-zlh79\") pod \"openshift-state-metrics-84bd68b896-wsr5g\" (UID: \"231fbb85-1b53-4342-99bc-a25f64972033\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" Jul 16 12:11:16.889140 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.888951 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42b9dc92-8c58-474e-aa83-806dee5ed2df-metrics-client-ca\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.889140 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.888973 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rm6s\" (UniqueName: \"kubernetes.io/projected/42b9dc92-8c58-474e-aa83-806dee5ed2df-kube-api-access-9rm6s\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.890019 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.889784 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/231fbb85-1b53-4342-99bc-a25f64972033-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-wsr5g\" (UID: \"231fbb85-1b53-4342-99bc-a25f64972033\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" Jul 16 12:11:16.891570 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.891548 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/231fbb85-1b53-4342-99bc-a25f64972033-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-wsr5g\" (UID: \"231fbb85-1b53-4342-99bc-a25f64972033\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" Jul 16 12:11:16.891570 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:11:16.891561 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac5634c_d4f8_4e17_a719_6a6dafff2d5a.slice/crio-2f018c5f47907b1aed690e052bc17cbf3dcf51cb8ca69585b757d49c2eccb4a6 WatchSource:0}: Error finding container 2f018c5f47907b1aed690e052bc17cbf3dcf51cb8ca69585b757d49c2eccb4a6: Status 404 returned error can't find the container with id 2f018c5f47907b1aed690e052bc17cbf3dcf51cb8ca69585b757d49c2eccb4a6 Jul 16 12:11:16.891778 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.891659 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/231fbb85-1b53-4342-99bc-a25f64972033-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-wsr5g\" (UID: \"231fbb85-1b53-4342-99bc-a25f64972033\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" Jul 16 12:11:16.928338 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.928317 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlh79\" (UniqueName: \"kubernetes.io/projected/231fbb85-1b53-4342-99bc-a25f64972033-kube-api-access-zlh79\") pod \"openshift-state-metrics-84bd68b896-wsr5g\" (UID: \"231fbb85-1b53-4342-99bc-a25f64972033\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" Jul 16 12:11:16.989484 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.989448 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42b9dc92-8c58-474e-aa83-806dee5ed2df-node-exporter-wtmp\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.989484 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.989486 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42b9dc92-8c58-474e-aa83-806dee5ed2df-sys\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.989758 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.989504 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42b9dc92-8c58-474e-aa83-806dee5ed2df-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.989758 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.989542 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42b9dc92-8c58-474e-aa83-806dee5ed2df-node-exporter-tls\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.989758 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.989573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42b9dc92-8c58-474e-aa83-806dee5ed2df-metrics-client-ca\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.989758 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.989581 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42b9dc92-8c58-474e-aa83-806dee5ed2df-sys\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.989758 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.989658 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42b9dc92-8c58-474e-aa83-806dee5ed2df-node-exporter-wtmp\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.989758 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.989724 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rm6s\" (UniqueName: \"kubernetes.io/projected/42b9dc92-8c58-474e-aa83-806dee5ed2df-kube-api-access-9rm6s\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.990069 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.989781 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42b9dc92-8c58-474e-aa83-806dee5ed2df-root\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.990069 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.989821 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42b9dc92-8c58-474e-aa83-806dee5ed2df-node-exporter-textfile\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.990069 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.989851 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42b9dc92-8c58-474e-aa83-806dee5ed2df-node-exporter-accelerators-collector-config\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.990069 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.989882 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42b9dc92-8c58-474e-aa83-806dee5ed2df-root\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.990264 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.990156 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42b9dc92-8c58-474e-aa83-806dee5ed2df-node-exporter-textfile\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.990321 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.990278 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42b9dc92-8c58-474e-aa83-806dee5ed2df-metrics-client-ca\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.990431 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.990404 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42b9dc92-8c58-474e-aa83-806dee5ed2df-node-exporter-accelerators-collector-config\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.991207 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.991185 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" Jul 16 12:11:16.991875 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.991853 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42b9dc92-8c58-474e-aa83-806dee5ed2df-node-exporter-tls\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:16.992294 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:16.992276 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42b9dc92-8c58-474e-aa83-806dee5ed2df-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:17.034416 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.034363 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-744f7666f4-wm8ww"] Jul 16 12:11:17.034416 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.034400 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rm6s\" (UniqueName: \"kubernetes.io/projected/42b9dc92-8c58-474e-aa83-806dee5ed2df-kube-api-access-9rm6s\") pod \"node-exporter-c2r95\" (UID: \"42b9dc92-8c58-474e-aa83-806dee5ed2df\") " pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:17.038986 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.038964 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.048804 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.048510 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 12:11:17.048804 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.048533 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 12:11:17.048804 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.048748 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 12:11:17.048977 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.048835 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-b64mj\"" Jul 16 12:11:17.048977 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.048836 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 12:11:17.048977 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.048910 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 12:11:17.049130 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.049041 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 12:11:17.049180 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.049152 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 12:11:17.062528 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.062508 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c2r95" Jul 16 12:11:17.073256 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.073202 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-744f7666f4-wm8ww"] Jul 16 12:11:17.074430 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:11:17.074403 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b9dc92_8c58_474e_aa83_806dee5ed2df.slice/crio-76d0bc68fa241d8c5675d92e272f3786e8f2a6748fbb33bac223a0a4c9006663 WatchSource:0}: Error finding container 76d0bc68fa241d8c5675d92e272f3786e8f2a6748fbb33bac223a0a4c9006663: Status 404 returned error can't find the container with id 76d0bc68fa241d8c5675d92e272f3786e8f2a6748fbb33bac223a0a4c9006663 Jul 16 12:11:17.173809 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.173729 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g"] Jul 16 12:11:17.176492 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:11:17.176465 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231fbb85_1b53_4342_99bc_a25f64972033.slice/crio-112a4d3c73afddd8b1c2f191e43517e3bb5540898ee68fdeda278bced749e5b0 WatchSource:0}: Error finding container 112a4d3c73afddd8b1c2f191e43517e3bb5540898ee68fdeda278bced749e5b0: Status 404 returned error can't find the container with id 112a4d3c73afddd8b1c2f191e43517e3bb5540898ee68fdeda278bced749e5b0 Jul 16 12:11:17.191188 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.191169 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-config\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.191265 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.191205 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jp49\" (UniqueName: \"kubernetes.io/projected/d2309fe5-e2c2-4140-a61c-076e740ae02d-kube-api-access-7jp49\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.191265 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.191255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-serving-cert\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.191355 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.191281 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-oauth-config\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.191389 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.191367 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-service-ca\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.191426 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.191393 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-oauth-serving-cert\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.292486 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.292459 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-config\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.292588 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.292501 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jp49\" (UniqueName: \"kubernetes.io/projected/d2309fe5-e2c2-4140-a61c-076e740ae02d-kube-api-access-7jp49\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.292588 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.292544 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-serving-cert\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.292588 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.292572 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-oauth-config\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.292754 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.292622 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-service-ca\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.292754 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.292645 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-oauth-serving-cert\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.293243 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.293217 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-config\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.293340 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.293322 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-oauth-serving-cert\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.293653 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.293627 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-service-ca\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.295653 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.295632 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-serving-cert\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.295752 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.295736 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-oauth-config\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.324128 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.324100 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jp49\" (UniqueName: \"kubernetes.io/projected/d2309fe5-e2c2-4140-a61c-076e740ae02d-kube-api-access-7jp49\") pod \"console-744f7666f4-wm8ww\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.324521 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.324317 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" event={"ID":"231fbb85-1b53-4342-99bc-a25f64972033","Type":"ContainerStarted","Data":"3b8d5e072beef2727c8afe97a2e98a9752921fbb0aecd77ddc9346d54e15d27b"} Jul 16 12:11:17.324521 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.324399 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" event={"ID":"231fbb85-1b53-4342-99bc-a25f64972033","Type":"ContainerStarted","Data":"9ccabfc8f4092f5bbb2fbd8eae7fac417fafa2966619da5132ac45b5c674877c"} Jul 16 12:11:17.324521 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.324458 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" event={"ID":"231fbb85-1b53-4342-99bc-a25f64972033","Type":"ContainerStarted","Data":"112a4d3c73afddd8b1c2f191e43517e3bb5540898ee68fdeda278bced749e5b0"} Jul 16 12:11:17.325568 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.325547 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" event={"ID":"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a","Type":"ContainerStarted","Data":"2f018c5f47907b1aed690e052bc17cbf3dcf51cb8ca69585b757d49c2eccb4a6"} Jul 16 12:11:17.326837 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.326814 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c2r95" event={"ID":"42b9dc92-8c58-474e-aa83-806dee5ed2df","Type":"ContainerStarted","Data":"76d0bc68fa241d8c5675d92e272f3786e8f2a6748fbb33bac223a0a4c9006663"} Jul 16 12:11:17.352426 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.352392 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:17.525421 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:17.525396 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-744f7666f4-wm8ww"] Jul 16 12:11:17.527923 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:11:17.527897 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2309fe5_e2c2_4140_a61c_076e740ae02d.slice/crio-15ac6a962a4688af2da31ad022d3b9d71850ebeedd1855e4e1de6917df4b11fc WatchSource:0}: Error finding container 15ac6a962a4688af2da31ad022d3b9d71850ebeedd1855e4e1de6917df4b11fc: Status 404 returned error can't find the container with id 15ac6a962a4688af2da31ad022d3b9d71850ebeedd1855e4e1de6917df4b11fc Jul 16 12:11:18.237051 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:18.236988 2559 patch_prober.go:28] interesting pod/image-registry-55bc8548d9-6d7s8 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 16 12:11:18.237225 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:18.237072 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" podUID="bf0147eb-963a-4fde-8d30-68fa336b7244" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 12:11:18.332425 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:18.332386 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744f7666f4-wm8ww" event={"ID":"d2309fe5-e2c2-4140-a61c-076e740ae02d","Type":"ContainerStarted","Data":"15ac6a962a4688af2da31ad022d3b9d71850ebeedd1855e4e1de6917df4b11fc"} Jul 16 12:11:18.334151 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:18.334122 2559 generic.go:356] "Generic (PLEG): container finished" podID="42b9dc92-8c58-474e-aa83-806dee5ed2df" containerID="a428ed36025255fc6a88ce1a051ee6bfda75c88c47b85e81e3ca09f1228da786" exitCode=0 Jul 16 12:11:18.334279 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:18.334207 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c2r95" event={"ID":"42b9dc92-8c58-474e-aa83-806dee5ed2df","Type":"ContainerDied","Data":"a428ed36025255fc6a88ce1a051ee6bfda75c88c47b85e81e3ca09f1228da786"} Jul 16 12:11:19.341605 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:19.341563 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c2r95" event={"ID":"42b9dc92-8c58-474e-aa83-806dee5ed2df","Type":"ContainerStarted","Data":"8ea51969da943aeb0e47157f10dcad953abb7205dbd11013d4bd122db0be945b"} Jul 16 12:11:19.342086 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:19.341632 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c2r95" event={"ID":"42b9dc92-8c58-474e-aa83-806dee5ed2df","Type":"ContainerStarted","Data":"eec6f014dd6b455ef4906deb1036af0b5e2404f883d28814132d79730850de0e"} Jul 16 12:11:19.343959 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:19.343923 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" event={"ID":"231fbb85-1b53-4342-99bc-a25f64972033","Type":"ContainerStarted","Data":"4f11c6a376ce9dc0c4addbbd0e29281530673c698698e4fcf3474e8c967819ed"} Jul 16 12:11:19.373306 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:19.373247 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c2r95" podStartSLOduration=2.352022346 podStartE2EDuration="3.373228221s" totalImagesPullingTime="949.531746ms" totalInitContainerRuntime="71.674129ms" isStatefulPod=true podCreationTimestamp="2026-07-16 12:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:11:19.372341396 +0000 UTC m=+257.522048853" watchObservedRunningTime="2026-07-16 12:11:19.373228221 +0000 UTC m=+257.522935655" Jul 16 12:11:19.454274 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:19.454221 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wsr5g" podStartSLOduration=2.410771023 podStartE2EDuration="3.45420568s" totalImagesPullingTime="1.043434657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:11:19.453892645 +0000 UTC m=+257.603600079" watchObservedRunningTime="2026-07-16 12:11:19.45420568 +0000 UTC m=+257.603913092" Jul 16 12:11:21.355426 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:21.355392 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744f7666f4-wm8ww" event={"ID":"d2309fe5-e2c2-4140-a61c-076e740ae02d","Type":"ContainerStarted","Data":"5fb812920e53c5f5e41b610493c94d117d0d9044b08722ddba2fb84c56ad1542"} Jul 16 12:11:21.383744 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:21.383692 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-744f7666f4-wm8ww" podStartSLOduration=2.234139115 podStartE2EDuration="5.38366098s" totalImagesPullingTime="3.149521865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:11:21.38266546 +0000 UTC m=+259.532372894" watchObservedRunningTime="2026-07-16 12:11:21.38366098 +0000 UTC m=+259.533368412" Jul 16 12:11:22.027382 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.027349 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-d5656d965-k22f9"] Jul 16 12:11:22.031105 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.031080 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.035420 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.035384 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 16 12:11:22.035585 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.035520 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 16 12:11:22.035585 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.035554 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-blqwv\"" Jul 16 12:11:22.035703 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.035593 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 16 12:11:22.035703 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.035658 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 16 12:11:22.035859 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.035847 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 16 12:11:22.041701 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.041664 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 16 12:11:22.063094 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.063059 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d5656d965-k22f9"] Jul 16 12:11:22.134465 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.134426 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b8049df4-0e4e-4afd-9888-b45d70b596a1-secret-telemeter-client\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.134624 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.134476 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8049df4-0e4e-4afd-9888-b45d70b596a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.134624 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.134552 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b8049df4-0e4e-4afd-9888-b45d70b596a1-telemeter-client-tls\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.134624 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.134586 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8049df4-0e4e-4afd-9888-b45d70b596a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.134624 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.134620 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b8049df4-0e4e-4afd-9888-b45d70b596a1-federate-client-tls\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.134836 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.134646 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zmcv\" (UniqueName: \"kubernetes.io/projected/b8049df4-0e4e-4afd-9888-b45d70b596a1-kube-api-access-6zmcv\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.134836 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.134694 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8049df4-0e4e-4afd-9888-b45d70b596a1-serving-certs-ca-bundle\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.134836 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.134723 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8049df4-0e4e-4afd-9888-b45d70b596a1-metrics-client-ca\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.235405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.235371 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b8049df4-0e4e-4afd-9888-b45d70b596a1-secret-telemeter-client\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.235578 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.235423 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8049df4-0e4e-4afd-9888-b45d70b596a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.235578 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.235455 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b8049df4-0e4e-4afd-9888-b45d70b596a1-telemeter-client-tls\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.235578 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.235475 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8049df4-0e4e-4afd-9888-b45d70b596a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.235578 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.235500 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b8049df4-0e4e-4afd-9888-b45d70b596a1-federate-client-tls\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.235578 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.235522 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zmcv\" (UniqueName: \"kubernetes.io/projected/b8049df4-0e4e-4afd-9888-b45d70b596a1-kube-api-access-6zmcv\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.235578 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.235550 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8049df4-0e4e-4afd-9888-b45d70b596a1-serving-certs-ca-bundle\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.235578 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.235575 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8049df4-0e4e-4afd-9888-b45d70b596a1-metrics-client-ca\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.236441 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.236416 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8049df4-0e4e-4afd-9888-b45d70b596a1-metrics-client-ca\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.236612 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.236586 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8049df4-0e4e-4afd-9888-b45d70b596a1-serving-certs-ca-bundle\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.236787 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.236759 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8049df4-0e4e-4afd-9888-b45d70b596a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.238328 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.238303 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b8049df4-0e4e-4afd-9888-b45d70b596a1-secret-telemeter-client\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.238398 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.238346 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b8049df4-0e4e-4afd-9888-b45d70b596a1-federate-client-tls\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.238517 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.238495 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8049df4-0e4e-4afd-9888-b45d70b596a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.238551 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.238514 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b8049df4-0e4e-4afd-9888-b45d70b596a1-telemeter-client-tls\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.247812 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.247780 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zmcv\" (UniqueName: \"kubernetes.io/projected/b8049df4-0e4e-4afd-9888-b45d70b596a1-kube-api-access-6zmcv\") pod \"telemeter-client-d5656d965-k22f9\" (UID: \"b8049df4-0e4e-4afd-9888-b45d70b596a1\") " pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.340663 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.340562 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" Jul 16 12:11:22.361174 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.361139 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" event={"ID":"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a","Type":"ContainerStarted","Data":"eb8a7cd7636da6b519c543c57c89eb44d3b9ea79d3792214bc15395acbb8a6ea"} Jul 16 12:11:22.361505 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.361217 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" event={"ID":"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a","Type":"ContainerStarted","Data":"c37e3fbf7b8625e0f6fc77696e6b6c56e35c0c81cd3dbbf214c184ad37b3d007"} Jul 16 12:11:22.361505 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.361246 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" event={"ID":"9ac5634c-d4f8-4e17-a719-6a6dafff2d5a","Type":"ContainerStarted","Data":"82d7c96336fd6dbd6df2a8c6478f562bd87eca04ffc0969034a12b011b9b1c95"} Jul 16 12:11:22.399950 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.399905 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-f2hjc" podStartSLOduration=1.575835543 podStartE2EDuration="6.399889013s" totalImagesPullingTime="4.82405347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:11:22.396897417 +0000 UTC m=+260.546604850" watchObservedRunningTime="2026-07-16 12:11:22.399889013 +0000 UTC m=+260.549596446" Jul 16 12:11:22.493148 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:22.493114 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d5656d965-k22f9"] Jul 16 12:11:22.496198 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:11:22.496174 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8049df4_0e4e_4afd_9888_b45d70b596a1.slice/crio-9606e269ea8421dcb6e3403ced8b682b7bbaa1ef8432eda4f36ca60570ad38b5 WatchSource:0}: Error finding container 9606e269ea8421dcb6e3403ced8b682b7bbaa1ef8432eda4f36ca60570ad38b5: Status 404 returned error can't find the container with id 9606e269ea8421dcb6e3403ced8b682b7bbaa1ef8432eda4f36ca60570ad38b5 Jul 16 12:11:23.182830 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.182790 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:11:23.187354 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.187322 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.191728 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.191703 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 12:11:23.191924 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.191902 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 12:11:23.192002 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.191947 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 12:11:23.192002 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.191992 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 12:11:23.192273 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.192258 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pkzpp\"" Jul 16 12:11:23.192443 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.192422 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 12:11:23.192443 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.192430 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3u7u1c05vgv2e\"" Jul 16 12:11:23.192443 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.192446 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 12:11:23.192638 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.192617 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 12:11:23.192712 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.192641 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 12:11:23.192814 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.192796 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 12:11:23.193036 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.193021 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 12:11:23.193693 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.193659 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 12:11:23.195552 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.195537 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 12:11:23.197883 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.197866 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 12:11:23.202447 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.202426 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 12:11:23.208694 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.208656 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:11:23.209701 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.209664 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 12:11:23.345176 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345143 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/137cb376-7287-48a3-83a2-660a869af737-config-out\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345333 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345204 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345333 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345258 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-config\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345333 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345281 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345333 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345305 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf9s4\" (UniqueName: \"kubernetes.io/projected/137cb376-7287-48a3-83a2-660a869af737-kube-api-access-wf9s4\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345490 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345362 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345490 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345393 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345490 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345410 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345490 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345429 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345490 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345451 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345645 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345499 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345645 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345538 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345645 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345572 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345645 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345590 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345645 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345612 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345645 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345640 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345851 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345658 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/137cb376-7287-48a3-83a2-660a869af737-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345851 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345694 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345851 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345717 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-web-config\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.345851 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.345765 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.366144 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.366112 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" event={"ID":"b8049df4-0e4e-4afd-9888-b45d70b596a1","Type":"ContainerStarted","Data":"9606e269ea8421dcb6e3403ced8b682b7bbaa1ef8432eda4f36ca60570ad38b5"} Jul 16 12:11:23.446640 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.446562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.446640 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.446600 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.446640 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.446615 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.446931 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.446692 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.446931 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.446742 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/137cb376-7287-48a3-83a2-660a869af737-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.446931 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.446769 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.446931 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.446792 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-web-config\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.446931 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.446854 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.446931 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.446885 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/137cb376-7287-48a3-83a2-660a869af737-config-out\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.447283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.446935 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.447283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.446978 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-config\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.447283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.447024 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.447283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.447050 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf9s4\" (UniqueName: \"kubernetes.io/projected/137cb376-7287-48a3-83a2-660a869af737-kube-api-access-wf9s4\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.447283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.447095 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.447283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.447121 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.447283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.447130 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.447283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.447146 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.447283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.447190 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.447283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.447233 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.447779 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.447289 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.447779 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.447336 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.447779 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.447752 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.448487 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.448459 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.448605 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.448543 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.448841 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.448815 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.449122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.449095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.449564 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.449541 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.450317 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.450290 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/137cb376-7287-48a3-83a2-660a869af737-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.450428 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.450406 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.451351 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.450745 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.451351 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.451244 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.451351 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.451301 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.451846 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.451803 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/137cb376-7287-48a3-83a2-660a869af737-config-out\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.451919 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.451873 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.452396 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.452371 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.452493 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.452383 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-config\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.452942 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.452924 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-web-config\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.453082 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.453065 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.454023 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.453995 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.462740 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.462722 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf9s4\" (UniqueName: \"kubernetes.io/projected/137cb376-7287-48a3-83a2-660a869af737-kube-api-access-wf9s4\") pod \"prometheus-k8s-0\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.499949 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.499911 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:23.688315 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:23.688277 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:11:23.689465 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:11:23.689433 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137cb376_7287_48a3_83a2_660a869af737.slice/crio-7a0f315c5a6c0a7d48acf38008ca4a9f4032a7afada542364caacc3e7b3b5e50 WatchSource:0}: Error finding container 7a0f315c5a6c0a7d48acf38008ca4a9f4032a7afada542364caacc3e7b3b5e50: Status 404 returned error can't find the container with id 7a0f315c5a6c0a7d48acf38008ca4a9f4032a7afada542364caacc3e7b3b5e50 Jul 16 12:11:24.372887 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:24.372856 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"137cb376-7287-48a3-83a2-660a869af737","Type":"ContainerStarted","Data":"7a0f315c5a6c0a7d48acf38008ca4a9f4032a7afada542364caacc3e7b3b5e50"} Jul 16 12:11:25.378242 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:25.378202 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" event={"ID":"b8049df4-0e4e-4afd-9888-b45d70b596a1","Type":"ContainerStarted","Data":"0694dd0ecdbf0941eafeda1343ce8d448a922a9ce649192befc4d1944f46a229"} Jul 16 12:11:25.378699 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:25.378273 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" event={"ID":"b8049df4-0e4e-4afd-9888-b45d70b596a1","Type":"ContainerStarted","Data":"d8169ec194dc52b15e1066d214a064fb721d7ad26a498d00a71713f6c350b4dc"} Jul 16 12:11:25.378699 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:25.378303 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" event={"ID":"b8049df4-0e4e-4afd-9888-b45d70b596a1","Type":"ContainerStarted","Data":"a1843c69317fce28bff9bb560fc0568e87da6ce2877a7394580df6b2adcd9d4e"} Jul 16 12:11:25.379744 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:25.379718 2559 generic.go:356] "Generic (PLEG): container finished" podID="137cb376-7287-48a3-83a2-660a869af737" containerID="6dc5a727bf4068a59ec0027857a0ef31217657163839f75bbe008f1acb54f813" exitCode=0 Jul 16 12:11:25.379818 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:25.379757 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"137cb376-7287-48a3-83a2-660a869af737","Type":"ContainerDied","Data":"6dc5a727bf4068a59ec0027857a0ef31217657163839f75bbe008f1acb54f813"} Jul 16 12:11:25.420208 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:25.420103 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-d5656d965-k22f9" podStartSLOduration=2.01645978 podStartE2EDuration="4.420086843s" totalImagesPullingTime="2.403627063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:11:25.41747946 +0000 UTC m=+263.567186895" watchObservedRunningTime="2026-07-16 12:11:25.420086843 +0000 UTC m=+263.569794276" Jul 16 12:11:27.352588 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:27.352549 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:27.353017 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:27.352624 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:27.354182 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:27.354147 2559 patch_prober.go:28] interesting pod/console-744f7666f4-wm8ww container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.18:8443/health\": dial tcp 10.133.0.18:8443: connect: connection refused" start-of-body= Jul 16 12:11:27.354309 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:27.354199 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-744f7666f4-wm8ww" podUID="d2309fe5-e2c2-4140-a61c-076e740ae02d" containerName="console" probeResult="failure" output="Get \"https://10.133.0.18:8443/health\": dial tcp 10.133.0.18:8443: connect: connection refused" Jul 16 12:11:28.233596 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:28.233559 2559 patch_prober.go:28] interesting pod/image-registry-55bc8548d9-6d7s8 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 16 12:11:28.233780 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:28.233621 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" podUID="bf0147eb-963a-4fde-8d30-68fa336b7244" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 12:11:29.398673 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:29.398589 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"137cb376-7287-48a3-83a2-660a869af737","Type":"ContainerStarted","Data":"7b6d5b4ca306536cfe7226a2fd4d48d6d9f9204b992d5ebcf3cefe25465e13fa"} Jul 16 12:11:29.398673 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:29.398657 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"137cb376-7287-48a3-83a2-660a869af737","Type":"ContainerStarted","Data":"70aaf5096bc13792130b99debcb980424b4574d5353628fa21c1cce120d98924"} Jul 16 12:11:31.409084 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:31.409048 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"137cb376-7287-48a3-83a2-660a869af737","Type":"ContainerStarted","Data":"91521c864310cecb6f0446911a4612c71b648c1182ce9ff82378b731bb8167e6"} Jul 16 12:11:31.409444 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:31.409112 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"137cb376-7287-48a3-83a2-660a869af737","Type":"ContainerStarted","Data":"e80d28f1ddae807348cff55f59fd2d50bec60cdf482524f309c1e404d7b3e8a8"} Jul 16 12:11:31.409444 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:31.409146 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"137cb376-7287-48a3-83a2-660a869af737","Type":"ContainerStarted","Data":"8b1b814874e32ae112c19595c128be18d247ec858a1845f2c0e8481d07e912cb"} Jul 16 12:11:31.409444 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:31.409179 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"137cb376-7287-48a3-83a2-660a869af737","Type":"ContainerStarted","Data":"b0a76b20241a427b6805ad459925df7f997b070d5a28ac9cb80527804b25f268"} Jul 16 12:11:31.463548 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:31.463407 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.787471136 podStartE2EDuration="8.463385825s" totalImagesPullingTime="6.642827652s" totalInitContainerRuntime="33.087037ms" isStatefulPod=false podCreationTimestamp="2026-07-16 12:11:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:11:31.461656767 +0000 UTC m=+269.611364190" watchObservedRunningTime="2026-07-16 12:11:31.463385825 +0000 UTC m=+269.613093258" Jul 16 12:11:31.542955 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:31.542922 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-744f7666f4-wm8ww"] Jul 16 12:11:33.248175 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.248128 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" podUID="bf0147eb-963a-4fde-8d30-68fa336b7244" containerName="registry" containerID="cri-o://114b245114c8a16d424d2e8593d0dee20eb89978dc0fbc24b8b7eaa1f4885b9c" gracePeriod=30 Jul 16 12:11:33.419933 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.419903 2559 generic.go:356] "Generic (PLEG): container finished" podID="bf0147eb-963a-4fde-8d30-68fa336b7244" containerID="114b245114c8a16d424d2e8593d0dee20eb89978dc0fbc24b8b7eaa1f4885b9c" exitCode=0 Jul 16 12:11:33.420064 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.419964 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" event={"ID":"bf0147eb-963a-4fde-8d30-68fa336b7244","Type":"ContainerDied","Data":"114b245114c8a16d424d2e8593d0dee20eb89978dc0fbc24b8b7eaa1f4885b9c"} Jul 16 12:11:33.492489 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.492467 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:11:33.500283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.500229 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:11:33.549864 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.549834 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls\") pod \"bf0147eb-963a-4fde-8d30-68fa336b7244\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " Jul 16 12:11:33.550038 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.549877 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7h7r\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-kube-api-access-g7h7r\") pod \"bf0147eb-963a-4fde-8d30-68fa336b7244\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " Jul 16 12:11:33.550038 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.549928 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf0147eb-963a-4fde-8d30-68fa336b7244-installation-pull-secrets\") pod \"bf0147eb-963a-4fde-8d30-68fa336b7244\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " Jul 16 12:11:33.550038 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.549958 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf0147eb-963a-4fde-8d30-68fa336b7244-image-registry-private-configuration\") pod \"bf0147eb-963a-4fde-8d30-68fa336b7244\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " Jul 16 12:11:33.550038 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.549996 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-bound-sa-token\") pod \"bf0147eb-963a-4fde-8d30-68fa336b7244\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " Jul 16 12:11:33.550038 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.550035 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf0147eb-963a-4fde-8d30-68fa336b7244-trusted-ca\") pod \"bf0147eb-963a-4fde-8d30-68fa336b7244\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " Jul 16 12:11:33.550303 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.550065 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-certificates\") pod \"bf0147eb-963a-4fde-8d30-68fa336b7244\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " Jul 16 12:11:33.550303 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.550137 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf0147eb-963a-4fde-8d30-68fa336b7244-ca-trust-extracted\") pod \"bf0147eb-963a-4fde-8d30-68fa336b7244\" (UID: \"bf0147eb-963a-4fde-8d30-68fa336b7244\") " Jul 16 12:11:33.550775 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.550513 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf0147eb-963a-4fde-8d30-68fa336b7244-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bf0147eb-963a-4fde-8d30-68fa336b7244" (UID: "bf0147eb-963a-4fde-8d30-68fa336b7244"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:11:33.550775 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.550667 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf0147eb-963a-4fde-8d30-68fa336b7244-trusted-ca\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:11:33.551842 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.551805 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bf0147eb-963a-4fde-8d30-68fa336b7244" (UID: "bf0147eb-963a-4fde-8d30-68fa336b7244"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:11:33.552756 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.552711 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bf0147eb-963a-4fde-8d30-68fa336b7244" (UID: "bf0147eb-963a-4fde-8d30-68fa336b7244"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:11:33.552861 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.552769 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf0147eb-963a-4fde-8d30-68fa336b7244-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bf0147eb-963a-4fde-8d30-68fa336b7244" (UID: "bf0147eb-963a-4fde-8d30-68fa336b7244"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:11:33.553187 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.553163 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bf0147eb-963a-4fde-8d30-68fa336b7244" (UID: "bf0147eb-963a-4fde-8d30-68fa336b7244"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:11:33.553267 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.553203 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf0147eb-963a-4fde-8d30-68fa336b7244-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bf0147eb-963a-4fde-8d30-68fa336b7244" (UID: "bf0147eb-963a-4fde-8d30-68fa336b7244"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:11:33.553501 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.553482 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-kube-api-access-g7h7r" (OuterVolumeSpecName: "kube-api-access-g7h7r") pod "bf0147eb-963a-4fde-8d30-68fa336b7244" (UID: "bf0147eb-963a-4fde-8d30-68fa336b7244"). InnerVolumeSpecName "kube-api-access-g7h7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:11:33.558829 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.558805 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0147eb-963a-4fde-8d30-68fa336b7244-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bf0147eb-963a-4fde-8d30-68fa336b7244" (UID: "bf0147eb-963a-4fde-8d30-68fa336b7244"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:11:33.651700 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.651633 2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf0147eb-963a-4fde-8d30-68fa336b7244-installation-pull-secrets\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:11:33.651700 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.651698 2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf0147eb-963a-4fde-8d30-68fa336b7244-image-registry-private-configuration\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:11:33.651700 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.651710 2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-bound-sa-token\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:11:33.651700 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.651719 2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-certificates\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:11:33.651975 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.651728 2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf0147eb-963a-4fde-8d30-68fa336b7244-ca-trust-extracted\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:11:33.651975 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.651737 2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-registry-tls\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:11:33.651975 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:33.651745 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7h7r\" (UniqueName: \"kubernetes.io/projected/bf0147eb-963a-4fde-8d30-68fa336b7244-kube-api-access-g7h7r\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:11:34.425289 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:34.425256 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" Jul 16 12:11:34.425738 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:34.425257 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bc8548d9-6d7s8" event={"ID":"bf0147eb-963a-4fde-8d30-68fa336b7244","Type":"ContainerDied","Data":"bf12c5df30571bdacbabf424620f5263c4acd9b3715b39f8a57ca07530ade1d4"} Jul 16 12:11:34.425738 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:34.425382 2559 scope.go:117] "RemoveContainer" containerID="114b245114c8a16d424d2e8593d0dee20eb89978dc0fbc24b8b7eaa1f4885b9c" Jul 16 12:11:34.454068 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:34.453970 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55bc8548d9-6d7s8"] Jul 16 12:11:34.455539 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:34.455518 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55bc8548d9-6d7s8"] Jul 16 12:11:34.556624 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:34.556587 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf0147eb-963a-4fde-8d30-68fa336b7244" path="/var/lib/kubelet/pods/bf0147eb-963a-4fde-8d30-68fa336b7244/volumes" Jul 16 12:11:44.022936 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:11:44.022889 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-f55c4" podUID="268f88f9-d545-4d19-9bc8-8695df1dc92c" Jul 16 12:11:44.462937 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:44.462859 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f55c4" Jul 16 12:11:47.168717 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:47.168592 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:11:47.171879 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:47.171846 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/268f88f9-d545-4d19-9bc8-8695df1dc92c-metrics-tls\") pod \"dns-default-f55c4\" (UID: \"268f88f9-d545-4d19-9bc8-8695df1dc92c\") " pod="openshift-dns/dns-default-f55c4" Jul 16 12:11:47.467097 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:47.467011 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v8966\"" Jul 16 12:11:47.474342 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:47.474313 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f55c4" Jul 16 12:11:47.477420 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:47.477380 2559 generic.go:356] "Generic (PLEG): container finished" podID="aa1dbbae-768e-4fae-bc87-177369420431" containerID="77186b665106e3a1d0327be5265301aebb9fe6966d98c296c3055557584ee8f3" exitCode=0 Jul 16 12:11:47.477475 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:47.477449 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l" event={"ID":"aa1dbbae-768e-4fae-bc87-177369420431","Type":"ContainerDied","Data":"77186b665106e3a1d0327be5265301aebb9fe6966d98c296c3055557584ee8f3"} Jul 16 12:11:47.477873 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:47.477856 2559 scope.go:117] "RemoveContainer" containerID="77186b665106e3a1d0327be5265301aebb9fe6966d98c296c3055557584ee8f3" Jul 16 12:11:47.633437 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:47.633399 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f55c4"] Jul 16 12:11:48.483824 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:48.483789 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-vdv2l" event={"ID":"aa1dbbae-768e-4fae-bc87-177369420431","Type":"ContainerStarted","Data":"59f46d4f1b76efe6fab17688187bb8742de2224f14e9b2e1e3e8041263178384"} Jul 16 12:11:48.485059 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:48.485030 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f55c4" event={"ID":"268f88f9-d545-4d19-9bc8-8695df1dc92c","Type":"ContainerStarted","Data":"3c0e684186e84445ad49bba5a5de7c36c14263345e92f64ce2367526f964d58e"} Jul 16 12:11:50.495119 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:50.495082 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f55c4" event={"ID":"268f88f9-d545-4d19-9bc8-8695df1dc92c","Type":"ContainerStarted","Data":"0ae11476ed9ce88cbc58abaedd039b1729a9d93fafabd89ab7f2d09e26c829f0"} Jul 16 12:11:50.495499 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:50.495136 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f55c4" event={"ID":"268f88f9-d545-4d19-9bc8-8695df1dc92c","Type":"ContainerStarted","Data":"a5691565f9327d21664763adfef2edb00fec08cd69d2d4ceb4fc2b19c93fc590"} Jul 16 12:11:50.495499 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:50.495175 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-f55c4" Jul 16 12:11:50.519109 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:50.519047 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-f55c4" podStartSLOduration=251.628186323 podStartE2EDuration="4m13.519025407s" totalImagesPullingTime="1.890839084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:11:50.518618331 +0000 UTC m=+288.668325765" watchObservedRunningTime="2026-07-16 12:11:50.519025407 +0000 UTC m=+288.668732845" Jul 16 12:11:56.563202 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:56.563142 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-744f7666f4-wm8ww" podUID="d2309fe5-e2c2-4140-a61c-076e740ae02d" containerName="console" containerID="cri-o://5fb812920e53c5f5e41b610493c94d117d0d9044b08722ddba2fb84c56ad1542" gracePeriod=15 Jul 16 12:11:56.834769 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:56.834671 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:56.954810 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:56.954766 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-oauth-config\") pod \"d2309fe5-e2c2-4140-a61c-076e740ae02d\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " Jul 16 12:11:56.954961 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:56.954832 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-config\") pod \"d2309fe5-e2c2-4140-a61c-076e740ae02d\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " Jul 16 12:11:56.954961 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:56.954879 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-oauth-serving-cert\") pod \"d2309fe5-e2c2-4140-a61c-076e740ae02d\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " Jul 16 12:11:56.954961 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:56.954897 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jp49\" (UniqueName: \"kubernetes.io/projected/d2309fe5-e2c2-4140-a61c-076e740ae02d-kube-api-access-7jp49\") pod \"d2309fe5-e2c2-4140-a61c-076e740ae02d\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " Jul 16 12:11:56.954961 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:56.954919 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-service-ca\") pod \"d2309fe5-e2c2-4140-a61c-076e740ae02d\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " Jul 16 12:11:56.955155 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:56.955111 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-serving-cert\") pod \"d2309fe5-e2c2-4140-a61c-076e740ae02d\" (UID: \"d2309fe5-e2c2-4140-a61c-076e740ae02d\") " Jul 16 12:11:56.955353 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:56.955322 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-service-ca" (OuterVolumeSpecName: "service-ca") pod "d2309fe5-e2c2-4140-a61c-076e740ae02d" (UID: "d2309fe5-e2c2-4140-a61c-076e740ae02d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:11:56.955353 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:56.955339 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d2309fe5-e2c2-4140-a61c-076e740ae02d" (UID: "d2309fe5-e2c2-4140-a61c-076e740ae02d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:11:56.955353 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:56.955325 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-config" (OuterVolumeSpecName: "console-config") pod "d2309fe5-e2c2-4140-a61c-076e740ae02d" (UID: "d2309fe5-e2c2-4140-a61c-076e740ae02d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:11:56.957298 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:56.957267 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d2309fe5-e2c2-4140-a61c-076e740ae02d" (UID: "d2309fe5-e2c2-4140-a61c-076e740ae02d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:11:56.957396 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:56.957290 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d2309fe5-e2c2-4140-a61c-076e740ae02d" (UID: "d2309fe5-e2c2-4140-a61c-076e740ae02d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:11:56.957396 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:56.957342 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2309fe5-e2c2-4140-a61c-076e740ae02d-kube-api-access-7jp49" (OuterVolumeSpecName: "kube-api-access-7jp49") pod "d2309fe5-e2c2-4140-a61c-076e740ae02d" (UID: "d2309fe5-e2c2-4140-a61c-076e740ae02d"). InnerVolumeSpecName "kube-api-access-7jp49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:11:57.055946 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:57.055911 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-oauth-serving-cert\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:11:57.055946 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:57.055939 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jp49\" (UniqueName: \"kubernetes.io/projected/d2309fe5-e2c2-4140-a61c-076e740ae02d-kube-api-access-7jp49\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:11:57.055946 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:57.055952 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-service-ca\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:11:57.056170 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:57.055961 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-serving-cert\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:11:57.056170 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:57.055970 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-oauth-config\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:11:57.056170 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:57.055978 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2309fe5-e2c2-4140-a61c-076e740ae02d-console-config\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:11:57.527323 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:57.527284 2559 generic.go:356] "Generic (PLEG): container finished" podID="d2309fe5-e2c2-4140-a61c-076e740ae02d" containerID="5fb812920e53c5f5e41b610493c94d117d0d9044b08722ddba2fb84c56ad1542" exitCode=0 Jul 16 12:11:57.527571 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:57.527332 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744f7666f4-wm8ww" event={"ID":"d2309fe5-e2c2-4140-a61c-076e740ae02d","Type":"ContainerDied","Data":"5fb812920e53c5f5e41b610493c94d117d0d9044b08722ddba2fb84c56ad1542"} Jul 16 12:11:57.527571 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:57.527342 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744f7666f4-wm8ww" Jul 16 12:11:57.527571 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:57.527383 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744f7666f4-wm8ww" event={"ID":"d2309fe5-e2c2-4140-a61c-076e740ae02d","Type":"ContainerDied","Data":"15ac6a962a4688af2da31ad022d3b9d71850ebeedd1855e4e1de6917df4b11fc"} Jul 16 12:11:57.527571 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:57.527410 2559 scope.go:117] "RemoveContainer" containerID="5fb812920e53c5f5e41b610493c94d117d0d9044b08722ddba2fb84c56ad1542" Jul 16 12:11:57.536603 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:57.536583 2559 scope.go:117] "RemoveContainer" containerID="5fb812920e53c5f5e41b610493c94d117d0d9044b08722ddba2fb84c56ad1542" Jul 16 12:11:57.536933 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:11:57.536906 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb812920e53c5f5e41b610493c94d117d0d9044b08722ddba2fb84c56ad1542\": container with ID starting with 5fb812920e53c5f5e41b610493c94d117d0d9044b08722ddba2fb84c56ad1542 not found: ID does not exist" containerID="5fb812920e53c5f5e41b610493c94d117d0d9044b08722ddba2fb84c56ad1542" Jul 16 12:11:57.537013 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:57.536944 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb812920e53c5f5e41b610493c94d117d0d9044b08722ddba2fb84c56ad1542"} err="failed to get container status \"5fb812920e53c5f5e41b610493c94d117d0d9044b08722ddba2fb84c56ad1542\": rpc error: code = NotFound desc = could not find container \"5fb812920e53c5f5e41b610493c94d117d0d9044b08722ddba2fb84c56ad1542\": container with ID starting with 5fb812920e53c5f5e41b610493c94d117d0d9044b08722ddba2fb84c56ad1542 not found: ID does not exist" Jul 16 12:11:57.559027 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:57.559000 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-744f7666f4-wm8ww"] Jul 16 12:11:57.569220 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:57.569194 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-744f7666f4-wm8ww"] Jul 16 12:11:58.556196 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:11:58.556163 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2309fe5-e2c2-4140-a61c-076e740ae02d" path="/var/lib/kubelet/pods/d2309fe5-e2c2-4140-a61c-076e740ae02d/volumes" Jul 16 12:12:00.502285 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:00.502252 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-f55c4" Jul 16 12:12:02.294848 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:02.294821 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:12:02.304756 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:02.304735 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:12:02.306060 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:02.306037 2559 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 12:12:09.671940 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:09.671908 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f55c4_268f88f9-d545-4d19-9bc8-8695df1dc92c/dns/0.log" Jul 16 12:12:09.874131 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:09.874100 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f55c4_268f88f9-d545-4d19-9bc8-8695df1dc92c/kube-rbac-proxy/0.log" Jul 16 12:12:10.869022 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:10.868981 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6vr5g_d1fe16f9-e90b-4dbf-8462-92f01545a15e/dns-node-resolver/0.log" Jul 16 12:12:12.062363 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:12.062331 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r4f95_94335d8a-acdd-4739-8741-ce365e66d8c5/serve-healthcheck-canary/0.log" Jul 16 12:12:12.269055 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:12.269025 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-65db858894-ckjk8_69e2cf70-9821-459a-a552-beb17a33e0e1/cluster-samples-operator/0.log" Jul 16 12:12:12.467469 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:12.467394 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-65db858894-ckjk8_69e2cf70-9821-459a-a552-beb17a33e0e1/cluster-samples-operator-watch/0.log" Jul 16 12:12:23.500700 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:23.500629 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:23.519311 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:23.519282 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:23.644204 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:23.644169 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:41.662367 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:41.662334 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:12:41.662917 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:41.662617 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="prometheus" containerID="cri-o://70aaf5096bc13792130b99debcb980424b4574d5353628fa21c1cce120d98924" gracePeriod=600 Jul 16 12:12:41.662917 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:41.662660 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="kube-rbac-proxy" containerID="cri-o://e80d28f1ddae807348cff55f59fd2d50bec60cdf482524f309c1e404d7b3e8a8" gracePeriod=600 Jul 16 12:12:41.662917 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:41.662716 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="kube-rbac-proxy-web" containerID="cri-o://8b1b814874e32ae112c19595c128be18d247ec858a1845f2c0e8481d07e912cb" gracePeriod=600 Jul 16 12:12:41.662917 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:41.662692 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="kube-rbac-proxy-thanos" containerID="cri-o://91521c864310cecb6f0446911a4612c71b648c1182ce9ff82378b731bb8167e6" gracePeriod=600 Jul 16 12:12:41.662917 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:41.662720 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="thanos-sidecar" containerID="cri-o://b0a76b20241a427b6805ad459925df7f997b070d5a28ac9cb80527804b25f268" gracePeriod=600 Jul 16 12:12:41.662917 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:41.662720 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="config-reloader" containerID="cri-o://7b6d5b4ca306536cfe7226a2fd4d48d6d9f9204b992d5ebcf3cefe25465e13fa" gracePeriod=600 Jul 16 12:12:42.711241 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:42.711191 2559 generic.go:356] "Generic (PLEG): container finished" podID="137cb376-7287-48a3-83a2-660a869af737" containerID="91521c864310cecb6f0446911a4612c71b648c1182ce9ff82378b731bb8167e6" exitCode=0 Jul 16 12:12:42.711241 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:42.711242 2559 generic.go:356] "Generic (PLEG): container finished" podID="137cb376-7287-48a3-83a2-660a869af737" containerID="e80d28f1ddae807348cff55f59fd2d50bec60cdf482524f309c1e404d7b3e8a8" exitCode=0 Jul 16 12:12:42.711615 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:42.711252 2559 generic.go:356] "Generic (PLEG): container finished" podID="137cb376-7287-48a3-83a2-660a869af737" containerID="b0a76b20241a427b6805ad459925df7f997b070d5a28ac9cb80527804b25f268" exitCode=0 Jul 16 12:12:42.711615 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:42.711263 2559 generic.go:356] "Generic (PLEG): container finished" podID="137cb376-7287-48a3-83a2-660a869af737" containerID="7b6d5b4ca306536cfe7226a2fd4d48d6d9f9204b992d5ebcf3cefe25465e13fa" exitCode=0 Jul 16 12:12:42.711615 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:42.711281 2559 generic.go:356] "Generic (PLEG): container finished" podID="137cb376-7287-48a3-83a2-660a869af737" containerID="70aaf5096bc13792130b99debcb980424b4574d5353628fa21c1cce120d98924" exitCode=0 Jul 16 12:12:42.711615 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:42.711256 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"137cb376-7287-48a3-83a2-660a869af737","Type":"ContainerDied","Data":"91521c864310cecb6f0446911a4612c71b648c1182ce9ff82378b731bb8167e6"} Jul 16 12:12:42.711615 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:42.711366 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"137cb376-7287-48a3-83a2-660a869af737","Type":"ContainerDied","Data":"e80d28f1ddae807348cff55f59fd2d50bec60cdf482524f309c1e404d7b3e8a8"} Jul 16 12:12:42.711615 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:42.711409 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"137cb376-7287-48a3-83a2-660a869af737","Type":"ContainerDied","Data":"b0a76b20241a427b6805ad459925df7f997b070d5a28ac9cb80527804b25f268"} Jul 16 12:12:42.711615 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:42.711448 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"137cb376-7287-48a3-83a2-660a869af737","Type":"ContainerDied","Data":"7b6d5b4ca306536cfe7226a2fd4d48d6d9f9204b992d5ebcf3cefe25465e13fa"} Jul 16 12:12:42.711615 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:42.711486 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"137cb376-7287-48a3-83a2-660a869af737","Type":"ContainerDied","Data":"70aaf5096bc13792130b99debcb980424b4574d5353628fa21c1cce120d98924"} Jul 16 12:12:42.919168 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:42.919138 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.050023 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.049975 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-config\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.050023 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.050023 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-kube-rbac-proxy\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.050296 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.050060 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/137cb376-7287-48a3-83a2-660a869af737-tls-assets\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.050296 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.050081 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-serving-certs-ca-bundle\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.050296 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.050109 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-metrics-client-ca\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.050296 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.050226 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf9s4\" (UniqueName: \"kubernetes.io/projected/137cb376-7287-48a3-83a2-660a869af737-kube-api-access-wf9s4\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.050296 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.050277 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.050562 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.050305 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-2\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.050562 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.050341 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.050788 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.050748 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:12:43.051058 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.051032 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:12:43.051137 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.051086 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-trusted-ca-bundle\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.051192 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.051141 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-kubelet-serving-ca-bundle\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.051192 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.051176 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-db\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.051362 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.051204 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-grpc-tls\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.051362 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.051253 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-metrics-client-certs\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.051457 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.051393 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-tls\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.051457 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.051422 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-web-config\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.051457 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.051425 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:12:43.051457 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.051455 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/137cb376-7287-48a3-83a2-660a869af737-config-out\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.051651 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.051501 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-0\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.051651 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.051530 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-1\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.051651 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.051565 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-thanos-prometheus-http-client-file\") pod \"137cb376-7287-48a3-83a2-660a869af737\" (UID: \"137cb376-7287-48a3-83a2-660a869af737\") " Jul 16 12:12:43.051848 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.051830 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.051903 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.051852 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-metrics-client-ca\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.051903 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.051866 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.052672 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.052540 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:12:43.052913 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.052889 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:12:43.053041 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.052921 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:12:43.053216 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.053189 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:12:43.053484 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.053465 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:12:43.053577 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.053527 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137cb376-7287-48a3-83a2-660a869af737-kube-api-access-wf9s4" (OuterVolumeSpecName: "kube-api-access-wf9s4") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "kube-api-access-wf9s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:12:43.053656 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.053592 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:12:43.053804 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.053787 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137cb376-7287-48a3-83a2-660a869af737-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:12:43.053926 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.053833 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:12:43.054880 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.054853 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-config" (OuterVolumeSpecName: "config") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:12:43.055309 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.055284 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:12:43.055692 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.055655 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:12:43.055883 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.055863 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137cb376-7287-48a3-83a2-660a869af737-config-out" (OuterVolumeSpecName: "config-out") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:12:43.055976 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.055879 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:12:43.055976 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.055917 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:12:43.056408 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.056382 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:12:43.065944 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.065913 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-web-config" (OuterVolumeSpecName: "web-config") pod "137cb376-7287-48a3-83a2-660a869af737" (UID: "137cb376-7287-48a3-83a2-660a869af737"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:12:43.152371 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.152331 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.152371 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.152364 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.152371 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.152379 2559 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.152646 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.152392 2559 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-config\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.152646 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.152404 2559 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-kube-rbac-proxy\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.152646 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.152416 2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/137cb376-7287-48a3-83a2-660a869af737-tls-assets\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.152646 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.152437 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf9s4\" (UniqueName: \"kubernetes.io/projected/137cb376-7287-48a3-83a2-660a869af737-kube-api-access-wf9s4\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.152646 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.152449 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.152646 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.152462 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.152646 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.152473 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.152646 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.152485 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137cb376-7287-48a3-83a2-660a869af737-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.152646 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.152497 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/137cb376-7287-48a3-83a2-660a869af737-prometheus-k8s-db\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.152646 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.152508 2559 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-grpc-tls\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.152646 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.152520 2559 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-metrics-client-certs\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.152646 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.152531 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.152646 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.152544 2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/137cb376-7287-48a3-83a2-660a869af737-web-config\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.152646 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.152556 2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/137cb376-7287-48a3-83a2-660a869af737-config-out\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:12:43.717964 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.717929 2559 generic.go:356] "Generic (PLEG): container finished" podID="137cb376-7287-48a3-83a2-660a869af737" containerID="8b1b814874e32ae112c19595c128be18d247ec858a1845f2c0e8481d07e912cb" exitCode=0 Jul 16 12:12:43.718384 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.718014 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.718384 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.718028 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"137cb376-7287-48a3-83a2-660a869af737","Type":"ContainerDied","Data":"8b1b814874e32ae112c19595c128be18d247ec858a1845f2c0e8481d07e912cb"} Jul 16 12:12:43.718384 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.718121 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"137cb376-7287-48a3-83a2-660a869af737","Type":"ContainerDied","Data":"7a0f315c5a6c0a7d48acf38008ca4a9f4032a7afada542364caacc3e7b3b5e50"} Jul 16 12:12:43.718384 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.718156 2559 scope.go:117] "RemoveContainer" containerID="91521c864310cecb6f0446911a4612c71b648c1182ce9ff82378b731bb8167e6" Jul 16 12:12:43.728641 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.728617 2559 scope.go:117] "RemoveContainer" containerID="e80d28f1ddae807348cff55f59fd2d50bec60cdf482524f309c1e404d7b3e8a8" Jul 16 12:12:43.742424 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.742399 2559 scope.go:117] "RemoveContainer" containerID="8b1b814874e32ae112c19595c128be18d247ec858a1845f2c0e8481d07e912cb" Jul 16 12:12:43.748564 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.748534 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:12:43.750537 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.750515 2559 scope.go:117] "RemoveContainer" containerID="b0a76b20241a427b6805ad459925df7f997b070d5a28ac9cb80527804b25f268" Jul 16 12:12:43.756292 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.756266 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:12:43.758923 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.758906 2559 scope.go:117] "RemoveContainer" containerID="7b6d5b4ca306536cfe7226a2fd4d48d6d9f9204b992d5ebcf3cefe25465e13fa" Jul 16 12:12:43.766124 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.766102 2559 scope.go:117] "RemoveContainer" containerID="70aaf5096bc13792130b99debcb980424b4574d5353628fa21c1cce120d98924" Jul 16 12:12:43.774061 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.773940 2559 scope.go:117] "RemoveContainer" containerID="6dc5a727bf4068a59ec0027857a0ef31217657163839f75bbe008f1acb54f813" Jul 16 12:12:43.781497 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.781374 2559 scope.go:117] "RemoveContainer" containerID="91521c864310cecb6f0446911a4612c71b648c1182ce9ff82378b731bb8167e6" Jul 16 12:12:43.781721 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:12:43.781692 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91521c864310cecb6f0446911a4612c71b648c1182ce9ff82378b731bb8167e6\": container with ID starting with 91521c864310cecb6f0446911a4612c71b648c1182ce9ff82378b731bb8167e6 not found: ID does not exist" containerID="91521c864310cecb6f0446911a4612c71b648c1182ce9ff82378b731bb8167e6" Jul 16 12:12:43.781825 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.781735 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91521c864310cecb6f0446911a4612c71b648c1182ce9ff82378b731bb8167e6"} err="failed to get container status \"91521c864310cecb6f0446911a4612c71b648c1182ce9ff82378b731bb8167e6\": rpc error: code = NotFound desc = could not find container \"91521c864310cecb6f0446911a4612c71b648c1182ce9ff82378b731bb8167e6\": container with ID starting with 91521c864310cecb6f0446911a4612c71b648c1182ce9ff82378b731bb8167e6 not found: ID does not exist" Jul 16 12:12:43.781825 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.781759 2559 scope.go:117] "RemoveContainer" containerID="e80d28f1ddae807348cff55f59fd2d50bec60cdf482524f309c1e404d7b3e8a8" Jul 16 12:12:43.782066 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:12:43.782031 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80d28f1ddae807348cff55f59fd2d50bec60cdf482524f309c1e404d7b3e8a8\": container with ID starting with e80d28f1ddae807348cff55f59fd2d50bec60cdf482524f309c1e404d7b3e8a8 not found: ID does not exist" containerID="e80d28f1ddae807348cff55f59fd2d50bec60cdf482524f309c1e404d7b3e8a8" Jul 16 12:12:43.782125 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.782074 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80d28f1ddae807348cff55f59fd2d50bec60cdf482524f309c1e404d7b3e8a8"} err="failed to get container status \"e80d28f1ddae807348cff55f59fd2d50bec60cdf482524f309c1e404d7b3e8a8\": rpc error: code = NotFound desc = could not find container \"e80d28f1ddae807348cff55f59fd2d50bec60cdf482524f309c1e404d7b3e8a8\": container with ID starting with e80d28f1ddae807348cff55f59fd2d50bec60cdf482524f309c1e404d7b3e8a8 not found: ID does not exist" Jul 16 12:12:43.782125 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.782092 2559 scope.go:117] "RemoveContainer" containerID="8b1b814874e32ae112c19595c128be18d247ec858a1845f2c0e8481d07e912cb" Jul 16 12:12:43.782369 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:12:43.782339 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1b814874e32ae112c19595c128be18d247ec858a1845f2c0e8481d07e912cb\": container with ID starting with 8b1b814874e32ae112c19595c128be18d247ec858a1845f2c0e8481d07e912cb not found: ID does not exist" containerID="8b1b814874e32ae112c19595c128be18d247ec858a1845f2c0e8481d07e912cb" Jul 16 12:12:43.782424 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.782366 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1b814874e32ae112c19595c128be18d247ec858a1845f2c0e8481d07e912cb"} err="failed to get container status \"8b1b814874e32ae112c19595c128be18d247ec858a1845f2c0e8481d07e912cb\": rpc error: code = NotFound desc = could not find container \"8b1b814874e32ae112c19595c128be18d247ec858a1845f2c0e8481d07e912cb\": container with ID starting with 8b1b814874e32ae112c19595c128be18d247ec858a1845f2c0e8481d07e912cb not found: ID does not exist" Jul 16 12:12:43.782424 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.782389 2559 scope.go:117] "RemoveContainer" containerID="b0a76b20241a427b6805ad459925df7f997b070d5a28ac9cb80527804b25f268" Jul 16 12:12:43.782633 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:12:43.782616 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a76b20241a427b6805ad459925df7f997b070d5a28ac9cb80527804b25f268\": container with ID starting with b0a76b20241a427b6805ad459925df7f997b070d5a28ac9cb80527804b25f268 not found: ID does not exist" containerID="b0a76b20241a427b6805ad459925df7f997b070d5a28ac9cb80527804b25f268" Jul 16 12:12:43.782704 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.782638 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a76b20241a427b6805ad459925df7f997b070d5a28ac9cb80527804b25f268"} err="failed to get container status \"b0a76b20241a427b6805ad459925df7f997b070d5a28ac9cb80527804b25f268\": rpc error: code = NotFound desc = could not find container \"b0a76b20241a427b6805ad459925df7f997b070d5a28ac9cb80527804b25f268\": container with ID starting with b0a76b20241a427b6805ad459925df7f997b070d5a28ac9cb80527804b25f268 not found: ID does not exist" Jul 16 12:12:43.782704 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.782652 2559 scope.go:117] "RemoveContainer" containerID="7b6d5b4ca306536cfe7226a2fd4d48d6d9f9204b992d5ebcf3cefe25465e13fa" Jul 16 12:12:43.783061 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:12:43.783019 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6d5b4ca306536cfe7226a2fd4d48d6d9f9204b992d5ebcf3cefe25465e13fa\": container with ID starting with 7b6d5b4ca306536cfe7226a2fd4d48d6d9f9204b992d5ebcf3cefe25465e13fa not found: ID does not exist" containerID="7b6d5b4ca306536cfe7226a2fd4d48d6d9f9204b992d5ebcf3cefe25465e13fa" Jul 16 12:12:43.783061 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.783053 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6d5b4ca306536cfe7226a2fd4d48d6d9f9204b992d5ebcf3cefe25465e13fa"} err="failed to get container status \"7b6d5b4ca306536cfe7226a2fd4d48d6d9f9204b992d5ebcf3cefe25465e13fa\": rpc error: code = NotFound desc = could not find container \"7b6d5b4ca306536cfe7226a2fd4d48d6d9f9204b992d5ebcf3cefe25465e13fa\": container with ID starting with 7b6d5b4ca306536cfe7226a2fd4d48d6d9f9204b992d5ebcf3cefe25465e13fa not found: ID does not exist" Jul 16 12:12:43.783221 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.783083 2559 scope.go:117] "RemoveContainer" containerID="70aaf5096bc13792130b99debcb980424b4574d5353628fa21c1cce120d98924" Jul 16 12:12:43.783422 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:12:43.783365 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70aaf5096bc13792130b99debcb980424b4574d5353628fa21c1cce120d98924\": container with ID starting with 70aaf5096bc13792130b99debcb980424b4574d5353628fa21c1cce120d98924 not found: ID does not exist" containerID="70aaf5096bc13792130b99debcb980424b4574d5353628fa21c1cce120d98924" Jul 16 12:12:43.783422 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.783393 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70aaf5096bc13792130b99debcb980424b4574d5353628fa21c1cce120d98924"} err="failed to get container status \"70aaf5096bc13792130b99debcb980424b4574d5353628fa21c1cce120d98924\": rpc error: code = NotFound desc = could not find container \"70aaf5096bc13792130b99debcb980424b4574d5353628fa21c1cce120d98924\": container with ID starting with 70aaf5096bc13792130b99debcb980424b4574d5353628fa21c1cce120d98924 not found: ID does not exist" Jul 16 12:12:43.783422 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.783412 2559 scope.go:117] "RemoveContainer" containerID="6dc5a727bf4068a59ec0027857a0ef31217657163839f75bbe008f1acb54f813" Jul 16 12:12:43.783747 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:12:43.783722 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc5a727bf4068a59ec0027857a0ef31217657163839f75bbe008f1acb54f813\": container with ID starting with 6dc5a727bf4068a59ec0027857a0ef31217657163839f75bbe008f1acb54f813 not found: ID does not exist" containerID="6dc5a727bf4068a59ec0027857a0ef31217657163839f75bbe008f1acb54f813" Jul 16 12:12:43.783833 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.783754 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc5a727bf4068a59ec0027857a0ef31217657163839f75bbe008f1acb54f813"} err="failed to get container status \"6dc5a727bf4068a59ec0027857a0ef31217657163839f75bbe008f1acb54f813\": rpc error: code = NotFound desc = could not find container \"6dc5a727bf4068a59ec0027857a0ef31217657163839f75bbe008f1acb54f813\": container with ID starting with 6dc5a727bf4068a59ec0027857a0ef31217657163839f75bbe008f1acb54f813 not found: ID does not exist" Jul 16 12:12:43.785182 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785163 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:12:43.785464 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785453 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="init-config-reloader" Jul 16 12:12:43.785506 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785466 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="init-config-reloader" Jul 16 12:12:43.785506 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785476 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="kube-rbac-proxy-web" Jul 16 12:12:43.785506 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785481 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="kube-rbac-proxy-web" Jul 16 12:12:43.785506 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785492 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="kube-rbac-proxy-thanos" Jul 16 12:12:43.785506 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785498 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="kube-rbac-proxy-thanos" Jul 16 12:12:43.785506 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785504 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="kube-rbac-proxy" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785510 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="kube-rbac-proxy" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785522 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="config-reloader" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785527 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="config-reloader" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785532 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf0147eb-963a-4fde-8d30-68fa336b7244" containerName="registry" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785537 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf0147eb-963a-4fde-8d30-68fa336b7244" containerName="registry" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785543 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="prometheus" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785548 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="prometheus" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785555 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2309fe5-e2c2-4140-a61c-076e740ae02d" containerName="console" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785559 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2309fe5-e2c2-4140-a61c-076e740ae02d" containerName="console" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785564 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="thanos-sidecar" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785568 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="thanos-sidecar" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785610 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2309fe5-e2c2-4140-a61c-076e740ae02d" containerName="console" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785618 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="config-reloader" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785624 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="kube-rbac-proxy-thanos" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785630 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="prometheus" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785636 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf0147eb-963a-4fde-8d30-68fa336b7244" containerName="registry" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785641 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="kube-rbac-proxy-web" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785647 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="thanos-sidecar" Jul 16 12:12:43.785690 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.785652 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="137cb376-7287-48a3-83a2-660a869af737" containerName="kube-rbac-proxy" Jul 16 12:12:43.790989 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.790966 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.793790 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.793761 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 12:12:43.794103 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.794085 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 12:12:43.794196 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.794107 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3u7u1c05vgv2e\"" Jul 16 12:12:43.794196 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.794161 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 12:12:43.794346 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.794196 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 12:12:43.794346 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.794244 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 12:12:43.794655 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.794639 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 12:12:43.794940 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.794921 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 12:12:43.794940 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.794935 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 12:12:43.795123 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.794941 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 12:12:43.795123 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.795059 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 12:12:43.795270 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.795136 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pkzpp\"" Jul 16 12:12:43.795334 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.795269 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 12:12:43.795937 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.795919 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 12:12:43.796041 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.795918 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 12:12:43.797963 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.797940 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 12:12:43.800415 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.800393 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 12:12:43.811869 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.811842 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:12:43.856594 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.856558 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7476a2bc-798c-445b-ba4f-f463fca083a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.856810 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.856603 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.856810 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.856640 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.856810 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.856662 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.856810 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.856705 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.856810 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.856731 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.856810 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.856754 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.856810 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.856778 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.856810 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.856801 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-config\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.857200 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.856837 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7476a2bc-798c-445b-ba4f-f463fca083a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.857200 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.856879 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zqg4\" (UniqueName: \"kubernetes.io/projected/7476a2bc-798c-445b-ba4f-f463fca083a3-kube-api-access-9zqg4\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.857200 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.856910 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.857200 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.856945 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.857200 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.856970 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.857200 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.856998 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7476a2bc-798c-445b-ba4f-f463fca083a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.857200 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.857111 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.857200 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.857141 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.857200 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.857159 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.857200 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.857183 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.857200 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.857198 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.957788 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.957739 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.957788 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.957794 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958019 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.957913 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958019 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.957957 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958109 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958084 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958150 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958121 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958196 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958152 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958196 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958181 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-config\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958290 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958205 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7476a2bc-798c-445b-ba4f-f463fca083a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958290 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958253 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zqg4\" (UniqueName: \"kubernetes.io/projected/7476a2bc-798c-445b-ba4f-f463fca083a3-kube-api-access-9zqg4\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958290 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958282 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958430 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958319 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958430 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958344 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958430 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958370 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7476a2bc-798c-445b-ba4f-f463fca083a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958430 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958406 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958638 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958439 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958638 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958463 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958638 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958500 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958638 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958531 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958638 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958564 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7476a2bc-798c-445b-ba4f-f463fca083a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.958638 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958600 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.959150 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.959120 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.961418 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.961085 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7476a2bc-798c-445b-ba4f-f463fca083a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.961418 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.961251 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.961418 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.961385 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.961418 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.961385 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.961709 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.961494 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.961709 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.961599 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.961709 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.958533 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.962433 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.962406 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-config\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.962535 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.962510 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.962733 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.962708 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7476a2bc-798c-445b-ba4f-f463fca083a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.963044 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.963020 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.963122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.963106 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.963518 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.963496 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7476a2bc-798c-445b-ba4f-f463fca083a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.963967 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.963938 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.964072 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.964018 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.964072 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.964029 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7476a2bc-798c-445b-ba4f-f463fca083a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.964287 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.964269 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7476a2bc-798c-445b-ba4f-f463fca083a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:43.967927 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:43.967908 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zqg4\" (UniqueName: \"kubernetes.io/projected/7476a2bc-798c-445b-ba4f-f463fca083a3-kube-api-access-9zqg4\") pod \"prometheus-k8s-0\" (UID: \"7476a2bc-798c-445b-ba4f-f463fca083a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:44.101864 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:44.101816 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:12:44.247306 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:44.247282 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:12:44.248767 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:12:44.248740 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7476a2bc_798c_445b_ba4f_f463fca083a3.slice/crio-ca19de178c5657cec34712301cba6857d667f44e0091003302f4e87807fa9493 WatchSource:0}: Error finding container ca19de178c5657cec34712301cba6857d667f44e0091003302f4e87807fa9493: Status 404 returned error can't find the container with id ca19de178c5657cec34712301cba6857d667f44e0091003302f4e87807fa9493 Jul 16 12:12:44.556074 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:44.556041 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="137cb376-7287-48a3-83a2-660a869af737" path="/var/lib/kubelet/pods/137cb376-7287-48a3-83a2-660a869af737/volumes" Jul 16 12:12:44.724243 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:44.724209 2559 generic.go:356] "Generic (PLEG): container finished" podID="7476a2bc-798c-445b-ba4f-f463fca083a3" containerID="c7dd44e156b216606fcc899a63e72d8c22c1f1d55db806639e9e26e5de6d4a3a" exitCode=0 Jul 16 12:12:44.724621 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:44.724264 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7476a2bc-798c-445b-ba4f-f463fca083a3","Type":"ContainerDied","Data":"c7dd44e156b216606fcc899a63e72d8c22c1f1d55db806639e9e26e5de6d4a3a"} Jul 16 12:12:44.724621 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:44.724312 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7476a2bc-798c-445b-ba4f-f463fca083a3","Type":"ContainerStarted","Data":"ca19de178c5657cec34712301cba6857d667f44e0091003302f4e87807fa9493"} Jul 16 12:12:45.731132 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:45.731098 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7476a2bc-798c-445b-ba4f-f463fca083a3","Type":"ContainerStarted","Data":"c017a09ed2209802909c0bbbf1ed34d17630d502644e5cd33ef09c406dfae5e2"} Jul 16 12:12:45.731547 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:45.731232 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7476a2bc-798c-445b-ba4f-f463fca083a3","Type":"ContainerStarted","Data":"156c89e7ed84fd21a79cb7de1ec99e8729f85382067b2d5278475fec4548969b"} Jul 16 12:12:45.731547 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:45.731280 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7476a2bc-798c-445b-ba4f-f463fca083a3","Type":"ContainerStarted","Data":"6776956c143c8a607c2253905dacabc4f24b7bf4dd996df712e1e3afa57f5156"} Jul 16 12:12:45.731547 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:45.731324 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7476a2bc-798c-445b-ba4f-f463fca083a3","Type":"ContainerStarted","Data":"b2f824349aff823220e1c9e6116a14708600ad347b6c6e6cbf3b73b9e85fd3b9"} Jul 16 12:12:45.731547 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:45.731364 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7476a2bc-798c-445b-ba4f-f463fca083a3","Type":"ContainerStarted","Data":"c8244bc17d3f548904998388c81edfdf1ec2dd1124b8a615ac1563c3cd4f9a3f"} Jul 16 12:12:45.731547 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:45.731401 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7476a2bc-798c-445b-ba4f-f463fca083a3","Type":"ContainerStarted","Data":"ed8fa5c87ae9d15ad14e757c152595f0240cccb7053d80c7fd4f5a59f0aff66f"} Jul 16 12:12:45.772656 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:45.772588 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.75217044 podStartE2EDuration="2.772567191s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.396751ms" isStatefulPod=false podCreationTimestamp="2026-07-16 12:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:12:45.770387847 +0000 UTC m=+343.920095279" watchObservedRunningTime="2026-07-16 12:12:45.772567191 +0000 UTC m=+343.922274625" Jul 16 12:12:49.102704 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:12:49.102643 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:13:44.102756 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:13:44.102709 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:13:44.118467 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:13:44.118435 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:13:44.980181 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:13:44.980153 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:14:23.539207 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:23.539169 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kf7hz"] Jul 16 12:14:23.542628 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:23.542605 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kf7hz" Jul 16 12:14:23.546891 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:23.546867 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 16 12:14:23.547041 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:23.546868 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 16 12:14:23.547846 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:23.547827 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-sv9jt\"" Jul 16 12:14:23.558038 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:23.558012 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kf7hz"] Jul 16 12:14:23.613753 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:23.613713 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9s2k\" (UniqueName: \"kubernetes.io/projected/69013c4b-f1c8-4285-bf20-90808cdbba31-kube-api-access-z9s2k\") pod \"cert-manager-webhook-6486ffd997-kf7hz\" (UID: \"69013c4b-f1c8-4285-bf20-90808cdbba31\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kf7hz" Jul 16 12:14:23.613943 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:23.613768 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69013c4b-f1c8-4285-bf20-90808cdbba31-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kf7hz\" (UID: \"69013c4b-f1c8-4285-bf20-90808cdbba31\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kf7hz" Jul 16 12:14:23.714923 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:23.714889 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9s2k\" (UniqueName: \"kubernetes.io/projected/69013c4b-f1c8-4285-bf20-90808cdbba31-kube-api-access-z9s2k\") pod \"cert-manager-webhook-6486ffd997-kf7hz\" (UID: \"69013c4b-f1c8-4285-bf20-90808cdbba31\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kf7hz" Jul 16 12:14:23.714923 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:23.714923 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69013c4b-f1c8-4285-bf20-90808cdbba31-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kf7hz\" (UID: \"69013c4b-f1c8-4285-bf20-90808cdbba31\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kf7hz" Jul 16 12:14:23.736378 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:23.736340 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69013c4b-f1c8-4285-bf20-90808cdbba31-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kf7hz\" (UID: \"69013c4b-f1c8-4285-bf20-90808cdbba31\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kf7hz" Jul 16 12:14:23.736547 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:23.736413 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9s2k\" (UniqueName: \"kubernetes.io/projected/69013c4b-f1c8-4285-bf20-90808cdbba31-kube-api-access-z9s2k\") pod \"cert-manager-webhook-6486ffd997-kf7hz\" (UID: \"69013c4b-f1c8-4285-bf20-90808cdbba31\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kf7hz" Jul 16 12:14:23.868602 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:23.868496 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kf7hz" Jul 16 12:14:24.020937 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:24.020900 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kf7hz"] Jul 16 12:14:24.030460 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:14:24.030427 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69013c4b_f1c8_4285_bf20_90808cdbba31.slice/crio-c175fe6476702cee87a432bb76432c71dfbb27360ab54dd937961320544cb183 WatchSource:0}: Error finding container c175fe6476702cee87a432bb76432c71dfbb27360ab54dd937961320544cb183: Status 404 returned error can't find the container with id c175fe6476702cee87a432bb76432c71dfbb27360ab54dd937961320544cb183 Jul 16 12:14:24.032301 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:24.032281 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:14:24.112757 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:24.112707 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kf7hz" event={"ID":"69013c4b-f1c8-4285-bf20-90808cdbba31","Type":"ContainerStarted","Data":"c175fe6476702cee87a432bb76432c71dfbb27360ab54dd937961320544cb183"} Jul 16 12:14:28.131533 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:28.131494 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kf7hz" event={"ID":"69013c4b-f1c8-4285-bf20-90808cdbba31","Type":"ContainerStarted","Data":"aa6658420af7845d0cd84de6a890eeb4f56a2a08a47987d7b108059d59f7ede6"} Jul 16 12:14:28.131943 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:28.131567 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kf7hz" Jul 16 12:14:28.157869 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:28.157812 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-kf7hz" podStartSLOduration=2.087738862 podStartE2EDuration="5.157794619s" totalImagesPullingTime="3.070055757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:14:28.157033784 +0000 UTC m=+446.306741246" watchObservedRunningTime="2026-07-16 12:14:28.157794619 +0000 UTC m=+446.307502051" Jul 16 12:14:34.137526 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:14:34.137492 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kf7hz" Jul 16 12:15:00.861030 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:00.860988 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw"] Jul 16 12:15:00.906111 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:00.906074 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" Jul 16 12:15:00.916316 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:00.916292 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:15:00.916956 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:00.916937 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-q44qf\"" Jul 16 12:15:00.917058 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:00.916964 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 16 12:15:00.917058 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:00.916944 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 16 12:15:00.917058 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:00.916938 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 16 12:15:00.930737 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:00.930662 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw"] Jul 16 12:15:00.943562 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:00.943534 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 16 12:15:01.031243 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.031209 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7319953c-b07a-41ad-a321-141905dce49a-metrics-cert\") pod \"lws-controller-manager-69c8d477dd-sdlvw\" (UID: \"7319953c-b07a-41ad-a321-141905dce49a\") " pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" Jul 16 12:15:01.031426 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.031257 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r7db\" (UniqueName: \"kubernetes.io/projected/7319953c-b07a-41ad-a321-141905dce49a-kube-api-access-4r7db\") pod \"lws-controller-manager-69c8d477dd-sdlvw\" (UID: \"7319953c-b07a-41ad-a321-141905dce49a\") " pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" Jul 16 12:15:01.031426 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.031287 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7319953c-b07a-41ad-a321-141905dce49a-cert\") pod \"lws-controller-manager-69c8d477dd-sdlvw\" (UID: \"7319953c-b07a-41ad-a321-141905dce49a\") " pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" Jul 16 12:15:01.031426 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.031337 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7319953c-b07a-41ad-a321-141905dce49a-manager-config\") pod \"lws-controller-manager-69c8d477dd-sdlvw\" (UID: \"7319953c-b07a-41ad-a321-141905dce49a\") " pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" Jul 16 12:15:01.132789 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.132698 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r7db\" (UniqueName: \"kubernetes.io/projected/7319953c-b07a-41ad-a321-141905dce49a-kube-api-access-4r7db\") pod \"lws-controller-manager-69c8d477dd-sdlvw\" (UID: \"7319953c-b07a-41ad-a321-141905dce49a\") " pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" Jul 16 12:15:01.132789 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.132739 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7319953c-b07a-41ad-a321-141905dce49a-cert\") pod \"lws-controller-manager-69c8d477dd-sdlvw\" (UID: \"7319953c-b07a-41ad-a321-141905dce49a\") " pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" Jul 16 12:15:01.132789 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.132761 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7319953c-b07a-41ad-a321-141905dce49a-manager-config\") pod \"lws-controller-manager-69c8d477dd-sdlvw\" (UID: \"7319953c-b07a-41ad-a321-141905dce49a\") " pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" Jul 16 12:15:01.133021 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.132800 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7319953c-b07a-41ad-a321-141905dce49a-metrics-cert\") pod \"lws-controller-manager-69c8d477dd-sdlvw\" (UID: \"7319953c-b07a-41ad-a321-141905dce49a\") " pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" Jul 16 12:15:01.133502 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.133471 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7319953c-b07a-41ad-a321-141905dce49a-manager-config\") pod \"lws-controller-manager-69c8d477dd-sdlvw\" (UID: \"7319953c-b07a-41ad-a321-141905dce49a\") " pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" Jul 16 12:15:01.135415 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.135394 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7319953c-b07a-41ad-a321-141905dce49a-metrics-cert\") pod \"lws-controller-manager-69c8d477dd-sdlvw\" (UID: \"7319953c-b07a-41ad-a321-141905dce49a\") " pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" Jul 16 12:15:01.135508 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.135446 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7319953c-b07a-41ad-a321-141905dce49a-cert\") pod \"lws-controller-manager-69c8d477dd-sdlvw\" (UID: \"7319953c-b07a-41ad-a321-141905dce49a\") " pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" Jul 16 12:15:01.153800 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.153767 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r7db\" (UniqueName: \"kubernetes.io/projected/7319953c-b07a-41ad-a321-141905dce49a-kube-api-access-4r7db\") pod \"lws-controller-manager-69c8d477dd-sdlvw\" (UID: \"7319953c-b07a-41ad-a321-141905dce49a\") " pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" Jul 16 12:15:01.216603 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.216566 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" Jul 16 12:15:01.340022 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.339989 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9"] Jul 16 12:15:01.343006 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.342955 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:01.347914 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.347887 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-m9js4\"" Jul 16 12:15:01.348043 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.347977 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 12:15:01.348977 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.348960 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 12:15:01.349281 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.349267 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 12:15:01.349332 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.349316 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 12:15:01.406210 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.406175 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw"] Jul 16 12:15:01.408846 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:15:01.408816 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7319953c_b07a_41ad_a321_141905dce49a.slice/crio-39a9258011d5ff66409cdc60f382c0b89c7c2cf8af5e4ef766ca6cf4d26f3801 WatchSource:0}: Error finding container 39a9258011d5ff66409cdc60f382c0b89c7c2cf8af5e4ef766ca6cf4d26f3801: Status 404 returned error can't find the container with id 39a9258011d5ff66409cdc60f382c0b89c7c2cf8af5e4ef766ca6cf4d26f3801 Jul 16 12:15:01.434546 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.434510 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7e3017a4-d440-47ff-b312-89920b129880-manifests\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-54dx9\" (UID: \"7e3017a4-d440-47ff-b312-89920b129880\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:01.434546 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.434549 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e3017a4-d440-47ff-b312-89920b129880-tmp\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-54dx9\" (UID: \"7e3017a4-d440-47ff-b312-89920b129880\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:01.434772 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.434573 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7e3017a4-d440-47ff-b312-89920b129880-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-54dx9\" (UID: \"7e3017a4-d440-47ff-b312-89920b129880\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:01.434772 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.434694 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ndgw\" (UniqueName: \"kubernetes.io/projected/7e3017a4-d440-47ff-b312-89920b129880-kube-api-access-9ndgw\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-54dx9\" (UID: \"7e3017a4-d440-47ff-b312-89920b129880\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:01.434772 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.434744 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e3017a4-d440-47ff-b312-89920b129880-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-54dx9\" (UID: \"7e3017a4-d440-47ff-b312-89920b129880\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:01.459243 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.459206 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9"] Jul 16 12:15:01.535513 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.535484 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7e3017a4-d440-47ff-b312-89920b129880-manifests\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-54dx9\" (UID: \"7e3017a4-d440-47ff-b312-89920b129880\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:01.535717 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.535523 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e3017a4-d440-47ff-b312-89920b129880-tmp\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-54dx9\" (UID: \"7e3017a4-d440-47ff-b312-89920b129880\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:01.535717 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.535548 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7e3017a4-d440-47ff-b312-89920b129880-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-54dx9\" (UID: \"7e3017a4-d440-47ff-b312-89920b129880\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:01.535835 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.535727 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ndgw\" (UniqueName: \"kubernetes.io/projected/7e3017a4-d440-47ff-b312-89920b129880-kube-api-access-9ndgw\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-54dx9\" (UID: \"7e3017a4-d440-47ff-b312-89920b129880\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:01.535835 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.535767 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e3017a4-d440-47ff-b312-89920b129880-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-54dx9\" (UID: \"7e3017a4-d440-47ff-b312-89920b129880\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:01.535944 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.535917 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7e3017a4-d440-47ff-b312-89920b129880-manifests\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-54dx9\" (UID: \"7e3017a4-d440-47ff-b312-89920b129880\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:01.538265 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.538237 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e3017a4-d440-47ff-b312-89920b129880-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-54dx9\" (UID: \"7e3017a4-d440-47ff-b312-89920b129880\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:01.538458 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.538437 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7e3017a4-d440-47ff-b312-89920b129880-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-54dx9\" (UID: \"7e3017a4-d440-47ff-b312-89920b129880\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:01.538943 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.538925 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e3017a4-d440-47ff-b312-89920b129880-tmp\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-54dx9\" (UID: \"7e3017a4-d440-47ff-b312-89920b129880\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:01.552175 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.552148 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ndgw\" (UniqueName: \"kubernetes.io/projected/7e3017a4-d440-47ff-b312-89920b129880-kube-api-access-9ndgw\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-54dx9\" (UID: \"7e3017a4-d440-47ff-b312-89920b129880\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:01.654716 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.654659 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:01.812670 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:01.812647 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9"] Jul 16 12:15:01.814840 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:15:01.814811 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3017a4_d440_47ff_b312_89920b129880.slice/crio-406a44718ec86e41c2290f352e885dda5e47bb7479b2acdec211429e0abcb0d4 WatchSource:0}: Error finding container 406a44718ec86e41c2290f352e885dda5e47bb7479b2acdec211429e0abcb0d4: Status 404 returned error can't find the container with id 406a44718ec86e41c2290f352e885dda5e47bb7479b2acdec211429e0abcb0d4 Jul 16 12:15:02.265476 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:02.265443 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" event={"ID":"7e3017a4-d440-47ff-b312-89920b129880","Type":"ContainerStarted","Data":"406a44718ec86e41c2290f352e885dda5e47bb7479b2acdec211429e0abcb0d4"} Jul 16 12:15:02.266645 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:02.266622 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" event={"ID":"7319953c-b07a-41ad-a321-141905dce49a","Type":"ContainerStarted","Data":"39a9258011d5ff66409cdc60f382c0b89c7c2cf8af5e4ef766ca6cf4d26f3801"} Jul 16 12:15:06.287838 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:06.287800 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" event={"ID":"7e3017a4-d440-47ff-b312-89920b129880","Type":"ContainerStarted","Data":"0a0bc99a407ed2de07b2aea74eeeb1f567427daed2449ff694acb7420f9a8edc"} Jul 16 12:15:06.289395 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:06.289371 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" event={"ID":"7319953c-b07a-41ad-a321-141905dce49a","Type":"ContainerStarted","Data":"250ab26049a750a682e77869ba34c016323bc80d1d81359570d24395f9946950"} Jul 16 12:15:06.289498 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:06.289430 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" Jul 16 12:15:06.348148 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:06.348095 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" podStartSLOduration=1.598157143 podStartE2EDuration="6.348079006s" totalImagesPullingTime="4.749921863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:15:06.347354527 +0000 UTC m=+484.497061993" watchObservedRunningTime="2026-07-16 12:15:06.348079006 +0000 UTC m=+484.497786439" Jul 16 12:15:07.295116 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:07.295084 2559 generic.go:356] "Generic (PLEG): container finished" podID="7e3017a4-d440-47ff-b312-89920b129880" containerID="0a0bc99a407ed2de07b2aea74eeeb1f567427daed2449ff694acb7420f9a8edc" exitCode=0 Jul 16 12:15:07.295512 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:07.295161 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" event={"ID":"7e3017a4-d440-47ff-b312-89920b129880","Type":"ContainerDied","Data":"0a0bc99a407ed2de07b2aea74eeeb1f567427daed2449ff694acb7420f9a8edc"} Jul 16 12:15:08.301791 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:08.301753 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" event={"ID":"7e3017a4-d440-47ff-b312-89920b129880","Type":"ContainerStarted","Data":"a12325b6f43b640dd5c64d37a8fa8332f4d5954eadcda4894c38c6085deebaf1"} Jul 16 12:15:08.302264 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:08.301840 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:08.335318 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:08.335267 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" podStartSLOduration=1.582942177 podStartE2EDuration="7.335251197s" totalImagesPullingTime="4.712578092s" totalInitContainerRuntime="1.039730928s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:15:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:15:08.329528569 +0000 UTC m=+486.479236004" watchObservedRunningTime="2026-07-16 12:15:08.335251197 +0000 UTC m=+486.484958630" Jul 16 12:15:17.297538 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:17.297505 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-69c8d477dd-sdlvw" Jul 16 12:15:19.308395 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:19.308364 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-54dx9" Jul 16 12:15:29.738439 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:29.738404 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-569f4f6548-cpsgs"] Jul 16 12:15:29.741777 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:29.741755 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-569f4f6548-cpsgs" Jul 16 12:15:29.749487 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:29.749465 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 16 12:15:29.749487 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:29.749479 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 12:15:29.749665 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:29.749556 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 12:15:29.750608 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:29.750588 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 16 12:15:29.750729 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:29.750695 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8hrh6\"" Jul 16 12:15:29.760006 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:29.759982 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-569f4f6548-cpsgs"] Jul 16 12:15:29.891757 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:29.891718 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c97c87ff-85f9-4b52-ab56-16d7dff6c46c-tmp\") pod \"kube-auth-proxy-569f4f6548-cpsgs\" (UID: \"c97c87ff-85f9-4b52-ab56-16d7dff6c46c\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-cpsgs" Jul 16 12:15:29.891924 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:29.891845 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c97c87ff-85f9-4b52-ab56-16d7dff6c46c-tls-certs\") pod \"kube-auth-proxy-569f4f6548-cpsgs\" (UID: \"c97c87ff-85f9-4b52-ab56-16d7dff6c46c\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-cpsgs" Jul 16 12:15:29.891966 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:29.891917 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9tkt\" (UniqueName: \"kubernetes.io/projected/c97c87ff-85f9-4b52-ab56-16d7dff6c46c-kube-api-access-j9tkt\") pod \"kube-auth-proxy-569f4f6548-cpsgs\" (UID: \"c97c87ff-85f9-4b52-ab56-16d7dff6c46c\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-cpsgs" Jul 16 12:15:29.993353 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:29.993275 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c97c87ff-85f9-4b52-ab56-16d7dff6c46c-tls-certs\") pod \"kube-auth-proxy-569f4f6548-cpsgs\" (UID: \"c97c87ff-85f9-4b52-ab56-16d7dff6c46c\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-cpsgs" Jul 16 12:15:29.993353 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:29.993321 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9tkt\" (UniqueName: \"kubernetes.io/projected/c97c87ff-85f9-4b52-ab56-16d7dff6c46c-kube-api-access-j9tkt\") pod \"kube-auth-proxy-569f4f6548-cpsgs\" (UID: \"c97c87ff-85f9-4b52-ab56-16d7dff6c46c\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-cpsgs" Jul 16 12:15:29.993548 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:29.993374 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c97c87ff-85f9-4b52-ab56-16d7dff6c46c-tmp\") pod \"kube-auth-proxy-569f4f6548-cpsgs\" (UID: \"c97c87ff-85f9-4b52-ab56-16d7dff6c46c\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-cpsgs" Jul 16 12:15:29.995554 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:29.995535 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c97c87ff-85f9-4b52-ab56-16d7dff6c46c-tmp\") pod \"kube-auth-proxy-569f4f6548-cpsgs\" (UID: \"c97c87ff-85f9-4b52-ab56-16d7dff6c46c\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-cpsgs" Jul 16 12:15:29.995824 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:29.995804 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c97c87ff-85f9-4b52-ab56-16d7dff6c46c-tls-certs\") pod \"kube-auth-proxy-569f4f6548-cpsgs\" (UID: \"c97c87ff-85f9-4b52-ab56-16d7dff6c46c\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-cpsgs" Jul 16 12:15:30.005810 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:30.005785 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9tkt\" (UniqueName: \"kubernetes.io/projected/c97c87ff-85f9-4b52-ab56-16d7dff6c46c-kube-api-access-j9tkt\") pod \"kube-auth-proxy-569f4f6548-cpsgs\" (UID: \"c97c87ff-85f9-4b52-ab56-16d7dff6c46c\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-cpsgs" Jul 16 12:15:30.051663 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:30.051628 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-569f4f6548-cpsgs" Jul 16 12:15:30.194335 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:15:30.194179 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97c87ff_85f9_4b52_ab56_16d7dff6c46c.slice/crio-60d854a1e07bf9cff2ddeac8430741ac1248076ef4b1fd3433f9ed8afbc858d7 WatchSource:0}: Error finding container 60d854a1e07bf9cff2ddeac8430741ac1248076ef4b1fd3433f9ed8afbc858d7: Status 404 returned error can't find the container with id 60d854a1e07bf9cff2ddeac8430741ac1248076ef4b1fd3433f9ed8afbc858d7 Jul 16 12:15:30.201732 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:30.201702 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-569f4f6548-cpsgs"] Jul 16 12:15:30.394549 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:30.394454 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-569f4f6548-cpsgs" event={"ID":"c97c87ff-85f9-4b52-ab56-16d7dff6c46c","Type":"ContainerStarted","Data":"60d854a1e07bf9cff2ddeac8430741ac1248076ef4b1fd3433f9ed8afbc858d7"} Jul 16 12:15:33.418380 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:33.418344 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-569f4f6548-cpsgs" event={"ID":"c97c87ff-85f9-4b52-ab56-16d7dff6c46c","Type":"ContainerStarted","Data":"a20107888babc88da5a13d53e2a9a8bddf0ae3d5ca8d7652c85670b2311f4d12"} Jul 16 12:15:33.441422 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:15:33.441312 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-569f4f6548-cpsgs" podStartSLOduration=1.452856084 podStartE2EDuration="4.441295805s" totalImagesPullingTime="2.988439721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:15:33.43884306 +0000 UTC m=+511.588550514" watchObservedRunningTime="2026-07-16 12:15:33.441295805 +0000 UTC m=+511.591003290" Jul 16 12:16:47.313734 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.313671 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85d67c5cb5-j46rk"] Jul 16 12:16:47.319087 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.319060 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.323065 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.323041 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 12:16:47.323065 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.323054 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-b64mj\"" Jul 16 12:16:47.323254 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.323041 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 12:16:47.323254 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.323093 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 12:16:47.323433 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.323415 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 12:16:47.323505 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.323437 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 12:16:47.323505 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.323417 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 12:16:47.327577 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.327558 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 12:16:47.329343 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.329322 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 12:16:47.337719 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.337692 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85d67c5cb5-j46rk"] Jul 16 12:16:47.471649 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.471609 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-oauth-serving-cert\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.471843 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.471659 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-trusted-ca-bundle\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.471843 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.471708 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw7l8\" (UniqueName: \"kubernetes.io/projected/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-kube-api-access-mw7l8\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.471843 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.471761 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-console-oauth-config\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.471843 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.471794 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-console-serving-cert\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.471843 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.471822 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-service-ca\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.471843 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.471841 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-console-config\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.577746 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.572886 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-oauth-serving-cert\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.577746 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.572947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-trusted-ca-bundle\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.577746 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.572978 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw7l8\" (UniqueName: \"kubernetes.io/projected/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-kube-api-access-mw7l8\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.577746 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.573024 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-console-oauth-config\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.577746 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.573069 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-console-serving-cert\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.577746 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.573110 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-service-ca\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.577746 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.573139 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-console-config\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.577746 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.573941 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-console-config\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.577746 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.574502 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-oauth-serving-cert\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.577746 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.575331 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-trusted-ca-bundle\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.579287 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.578669 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-console-serving-cert\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.579287 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.579237 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-service-ca\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.589205 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.589177 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-console-oauth-config\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.591744 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.591716 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw7l8\" (UniqueName: \"kubernetes.io/projected/a5b88bcc-733a-4ac9-9a2a-691b09afab1a-kube-api-access-mw7l8\") pod \"console-85d67c5cb5-j46rk\" (UID: \"a5b88bcc-733a-4ac9-9a2a-691b09afab1a\") " pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.628977 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.628700 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:47.773105 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:47.773071 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85d67c5cb5-j46rk"] Jul 16 12:16:47.776547 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:16:47.776516 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b88bcc_733a_4ac9_9a2a_691b09afab1a.slice/crio-ee3d767a47b17387821a9a276d2cd65fc0bec84255633e566a2016026f40c2d9 WatchSource:0}: Error finding container ee3d767a47b17387821a9a276d2cd65fc0bec84255633e566a2016026f40c2d9: Status 404 returned error can't find the container with id ee3d767a47b17387821a9a276d2cd65fc0bec84255633e566a2016026f40c2d9 Jul 16 12:16:48.750892 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:48.750857 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d67c5cb5-j46rk" event={"ID":"a5b88bcc-733a-4ac9-9a2a-691b09afab1a","Type":"ContainerStarted","Data":"b8041e6d42eb5a41d1f6db47b678d03a793c5ac6dc291aaca3eca84ad5ef6ea2"} Jul 16 12:16:48.751358 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:48.750918 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d67c5cb5-j46rk" event={"ID":"a5b88bcc-733a-4ac9-9a2a-691b09afab1a","Type":"ContainerStarted","Data":"ee3d767a47b17387821a9a276d2cd65fc0bec84255633e566a2016026f40c2d9"} Jul 16 12:16:48.778586 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:48.778522 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85d67c5cb5-j46rk" podStartSLOduration=1.778505775 podStartE2EDuration="1.778505775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:16:48.775958701 +0000 UTC m=+586.925666136" watchObservedRunningTime="2026-07-16 12:16:48.778505775 +0000 UTC m=+586.928213208" Jul 16 12:16:57.629305 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:57.629245 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:57.629735 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:57.629361 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:57.634261 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:57.634224 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:16:57.791799 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:16:57.791767 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85d67c5cb5-j46rk" Jul 16 12:17:02.333170 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:02.333140 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:17:02.337309 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:02.337283 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:17:13.454957 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:13.454916 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml"] Jul 16 12:17:13.458427 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:13.458404 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" Jul 16 12:17:13.461385 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:13.461359 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 12:17:13.461524 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:13.461392 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-w7v5k\"" Jul 16 12:17:13.461524 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:13.461456 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 12:17:13.476374 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:13.476340 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml"] Jul 16 12:17:13.503272 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:13.503238 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ce0b580-3ae9-47f9-9ae3-f15b51ec697d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cmrml\" (UID: \"1ce0b580-3ae9-47f9-9ae3-f15b51ec697d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" Jul 16 12:17:13.503454 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:13.503301 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tmnf\" (UniqueName: \"kubernetes.io/projected/1ce0b580-3ae9-47f9-9ae3-f15b51ec697d-kube-api-access-7tmnf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cmrml\" (UID: \"1ce0b580-3ae9-47f9-9ae3-f15b51ec697d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" Jul 16 12:17:13.604347 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:13.604309 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ce0b580-3ae9-47f9-9ae3-f15b51ec697d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cmrml\" (UID: \"1ce0b580-3ae9-47f9-9ae3-f15b51ec697d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" Jul 16 12:17:13.604347 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:13.604366 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tmnf\" (UniqueName: \"kubernetes.io/projected/1ce0b580-3ae9-47f9-9ae3-f15b51ec697d-kube-api-access-7tmnf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cmrml\" (UID: \"1ce0b580-3ae9-47f9-9ae3-f15b51ec697d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" Jul 16 12:17:13.604791 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:13.604773 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ce0b580-3ae9-47f9-9ae3-f15b51ec697d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cmrml\" (UID: \"1ce0b580-3ae9-47f9-9ae3-f15b51ec697d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" Jul 16 12:17:13.619080 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:13.619045 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tmnf\" (UniqueName: \"kubernetes.io/projected/1ce0b580-3ae9-47f9-9ae3-f15b51ec697d-kube-api-access-7tmnf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cmrml\" (UID: \"1ce0b580-3ae9-47f9-9ae3-f15b51ec697d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" Jul 16 12:17:13.769435 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:13.769401 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" Jul 16 12:17:13.906288 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:13.906262 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml"] Jul 16 12:17:13.908521 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:17:13.908492 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce0b580_3ae9_47f9_9ae3_f15b51ec697d.slice/crio-f86fc40094c226f1e3e8dfdada041bedd7cfe429b6fe2dd406551a8955f77c99 WatchSource:0}: Error finding container f86fc40094c226f1e3e8dfdada041bedd7cfe429b6fe2dd406551a8955f77c99: Status 404 returned error can't find the container with id f86fc40094c226f1e3e8dfdada041bedd7cfe429b6fe2dd406551a8955f77c99 Jul 16 12:17:14.862965 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:14.862931 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" event={"ID":"1ce0b580-3ae9-47f9-9ae3-f15b51ec697d","Type":"ContainerStarted","Data":"f86fc40094c226f1e3e8dfdada041bedd7cfe429b6fe2dd406551a8955f77c99"} Jul 16 12:17:20.899936 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:20.899887 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" event={"ID":"1ce0b580-3ae9-47f9-9ae3-f15b51ec697d","Type":"ContainerStarted","Data":"314454bd8098f40edf29212246aa46f7473318a477240829b50fb18142b64696"} Jul 16 12:17:20.900402 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:20.900042 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" Jul 16 12:17:31.906726 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:31.906667 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" Jul 16 12:17:31.933535 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:31.933477 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" podStartSLOduration=12.697367845 podStartE2EDuration="18.933461114s" totalImagesPullingTime="6.236093269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:17:20.93326293 +0000 UTC m=+619.082970364" watchObservedRunningTime="2026-07-16 12:17:31.933461114 +0000 UTC m=+630.083168547" Jul 16 12:17:34.004252 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.004210 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml"] Jul 16 12:17:34.004718 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.004418 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" podUID="1ce0b580-3ae9-47f9-9ae3-f15b51ec697d" containerName="manager" containerID="cri-o://314454bd8098f40edf29212246aa46f7473318a477240829b50fb18142b64696" gracePeriod=2 Jul 16 12:17:34.013284 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.013255 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml"] Jul 16 12:17:34.045020 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.044990 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl"] Jul 16 12:17:34.045330 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.045318 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ce0b580-3ae9-47f9-9ae3-f15b51ec697d" containerName="manager" Jul 16 12:17:34.045382 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.045331 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ce0b580-3ae9-47f9-9ae3-f15b51ec697d" containerName="manager" Jul 16 12:17:34.045418 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.045396 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ce0b580-3ae9-47f9-9ae3-f15b51ec697d" containerName="manager" Jul 16 12:17:34.048473 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.048447 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" Jul 16 12:17:34.057712 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:17:34.056103 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cmrml\" is forbidden: User \"system:node:ip-10-0-143-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-101.ec2.internal' and this object" podUID="1ce0b580-3ae9-47f9-9ae3-f15b51ec697d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" Jul 16 12:17:34.079555 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.079523 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl"] Jul 16 12:17:34.195380 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.195342 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bac970c7-8ff5-4d23-b93c-54d231fa2b08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vb7nl\" (UID: \"bac970c7-8ff5-4d23-b93c-54d231fa2b08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" Jul 16 12:17:34.195530 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.195449 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86d4q\" (UniqueName: \"kubernetes.io/projected/bac970c7-8ff5-4d23-b93c-54d231fa2b08-kube-api-access-86d4q\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vb7nl\" (UID: \"bac970c7-8ff5-4d23-b93c-54d231fa2b08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" Jul 16 12:17:34.253273 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.253244 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" Jul 16 12:17:34.272532 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:17:34.272453 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cmrml\" is forbidden: User \"system:node:ip-10-0-143-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-101.ec2.internal' and this object" podUID="1ce0b580-3ae9-47f9-9ae3-f15b51ec697d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" Jul 16 12:17:34.296906 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.296875 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86d4q\" (UniqueName: \"kubernetes.io/projected/bac970c7-8ff5-4d23-b93c-54d231fa2b08-kube-api-access-86d4q\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vb7nl\" (UID: \"bac970c7-8ff5-4d23-b93c-54d231fa2b08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" Jul 16 12:17:34.297002 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.296926 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bac970c7-8ff5-4d23-b93c-54d231fa2b08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vb7nl\" (UID: \"bac970c7-8ff5-4d23-b93c-54d231fa2b08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" Jul 16 12:17:34.297333 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.297311 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bac970c7-8ff5-4d23-b93c-54d231fa2b08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vb7nl\" (UID: \"bac970c7-8ff5-4d23-b93c-54d231fa2b08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" Jul 16 12:17:34.327707 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.327641 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86d4q\" (UniqueName: \"kubernetes.io/projected/bac970c7-8ff5-4d23-b93c-54d231fa2b08-kube-api-access-86d4q\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vb7nl\" (UID: \"bac970c7-8ff5-4d23-b93c-54d231fa2b08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" Jul 16 12:17:34.388149 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.388117 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" Jul 16 12:17:34.397384 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.397352 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ce0b580-3ae9-47f9-9ae3-f15b51ec697d-extensions-socket-volume\") pod \"1ce0b580-3ae9-47f9-9ae3-f15b51ec697d\" (UID: \"1ce0b580-3ae9-47f9-9ae3-f15b51ec697d\") " Jul 16 12:17:34.397498 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.397413 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tmnf\" (UniqueName: \"kubernetes.io/projected/1ce0b580-3ae9-47f9-9ae3-f15b51ec697d-kube-api-access-7tmnf\") pod \"1ce0b580-3ae9-47f9-9ae3-f15b51ec697d\" (UID: \"1ce0b580-3ae9-47f9-9ae3-f15b51ec697d\") " Jul 16 12:17:34.397977 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.397944 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce0b580-3ae9-47f9-9ae3-f15b51ec697d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1ce0b580-3ae9-47f9-9ae3-f15b51ec697d" (UID: "1ce0b580-3ae9-47f9-9ae3-f15b51ec697d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:17:34.399791 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.399756 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce0b580-3ae9-47f9-9ae3-f15b51ec697d-kube-api-access-7tmnf" (OuterVolumeSpecName: "kube-api-access-7tmnf") pod "1ce0b580-3ae9-47f9-9ae3-f15b51ec697d" (UID: "1ce0b580-3ae9-47f9-9ae3-f15b51ec697d"). InnerVolumeSpecName "kube-api-access-7tmnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:17:34.498269 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.498235 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ce0b580-3ae9-47f9-9ae3-f15b51ec697d-extensions-socket-volume\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:17:34.498269 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.498268 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tmnf\" (UniqueName: \"kubernetes.io/projected/1ce0b580-3ae9-47f9-9ae3-f15b51ec697d-kube-api-access-7tmnf\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:17:34.547387 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.547303 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl"] Jul 16 12:17:34.550842 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:17:34.550813 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac970c7_8ff5_4d23_b93c_54d231fa2b08.slice/crio-56051fe5e46da1361769fd5f6f7aa6fbe8f8cbf28d02b57fdaa1ef2ea14c2291 WatchSource:0}: Error finding container 56051fe5e46da1361769fd5f6f7aa6fbe8f8cbf28d02b57fdaa1ef2ea14c2291: Status 404 returned error can't find the container with id 56051fe5e46da1361769fd5f6f7aa6fbe8f8cbf28d02b57fdaa1ef2ea14c2291 Jul 16 12:17:34.556398 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.556372 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ce0b580-3ae9-47f9-9ae3-f15b51ec697d" path="/var/lib/kubelet/pods/1ce0b580-3ae9-47f9-9ae3-f15b51ec697d/volumes" Jul 16 12:17:34.970553 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.970457 2559 generic.go:356] "Generic (PLEG): container finished" podID="1ce0b580-3ae9-47f9-9ae3-f15b51ec697d" containerID="314454bd8098f40edf29212246aa46f7473318a477240829b50fb18142b64696" exitCode=0 Jul 16 12:17:34.970553 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.970498 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmrml" Jul 16 12:17:34.970803 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.970556 2559 scope.go:117] "RemoveContainer" containerID="314454bd8098f40edf29212246aa46f7473318a477240829b50fb18142b64696" Jul 16 12:17:34.972475 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.972447 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" event={"ID":"bac970c7-8ff5-4d23-b93c-54d231fa2b08","Type":"ContainerStarted","Data":"d40a96d90b74ab43561a7823bed54fa6f41bbd95fa827bb414603adde01357f3"} Jul 16 12:17:34.972598 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.972507 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" event={"ID":"bac970c7-8ff5-4d23-b93c-54d231fa2b08","Type":"ContainerStarted","Data":"56051fe5e46da1361769fd5f6f7aa6fbe8f8cbf28d02b57fdaa1ef2ea14c2291"} Jul 16 12:17:34.979750 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.979730 2559 scope.go:117] "RemoveContainer" containerID="314454bd8098f40edf29212246aa46f7473318a477240829b50fb18142b64696" Jul 16 12:17:34.980107 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:17:34.980080 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314454bd8098f40edf29212246aa46f7473318a477240829b50fb18142b64696\": container with ID starting with 314454bd8098f40edf29212246aa46f7473318a477240829b50fb18142b64696 not found: ID does not exist" containerID="314454bd8098f40edf29212246aa46f7473318a477240829b50fb18142b64696" Jul 16 12:17:34.980198 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:34.980118 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314454bd8098f40edf29212246aa46f7473318a477240829b50fb18142b64696"} err="failed to get container status \"314454bd8098f40edf29212246aa46f7473318a477240829b50fb18142b64696\": rpc error: code = NotFound desc = could not find container \"314454bd8098f40edf29212246aa46f7473318a477240829b50fb18142b64696\": container with ID starting with 314454bd8098f40edf29212246aa46f7473318a477240829b50fb18142b64696 not found: ID does not exist" Jul 16 12:17:35.003734 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:35.003658 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" podStartSLOduration=1.003639489 podStartE2EDuration="1.003639489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:17:35.000537733 +0000 UTC m=+633.150245193" watchObservedRunningTime="2026-07-16 12:17:35.003639489 +0000 UTC m=+633.153347029" Jul 16 12:17:35.978952 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:35.978922 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" Jul 16 12:17:46.985576 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:17:46.985538 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" Jul 16 12:18:04.361215 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.361180 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl"] Jul 16 12:18:04.361648 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.361386 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" podUID="bac970c7-8ff5-4d23-b93c-54d231fa2b08" containerName="manager" containerID="cri-o://d40a96d90b74ab43561a7823bed54fa6f41bbd95fa827bb414603adde01357f3" gracePeriod=10 Jul 16 12:18:04.608035 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.608004 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" Jul 16 12:18:04.621791 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.621710 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s"] Jul 16 12:18:04.622128 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.622112 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac970c7-8ff5-4d23-b93c-54d231fa2b08" containerName="manager" Jul 16 12:18:04.622187 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.622133 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac970c7-8ff5-4d23-b93c-54d231fa2b08" containerName="manager" Jul 16 12:18:04.622232 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.622222 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac970c7-8ff5-4d23-b93c-54d231fa2b08" containerName="manager" Jul 16 12:18:04.625049 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.625034 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" Jul 16 12:18:04.641361 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.641330 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s"] Jul 16 12:18:04.655874 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.655843 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bac970c7-8ff5-4d23-b93c-54d231fa2b08-extensions-socket-volume\") pod \"bac970c7-8ff5-4d23-b93c-54d231fa2b08\" (UID: \"bac970c7-8ff5-4d23-b93c-54d231fa2b08\") " Jul 16 12:18:04.656036 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.655909 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86d4q\" (UniqueName: \"kubernetes.io/projected/bac970c7-8ff5-4d23-b93c-54d231fa2b08-kube-api-access-86d4q\") pod \"bac970c7-8ff5-4d23-b93c-54d231fa2b08\" (UID: \"bac970c7-8ff5-4d23-b93c-54d231fa2b08\") " Jul 16 12:18:04.656174 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.656133 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e86ef8d1-534c-4d8d-b07a-c5592c8311db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jng5s\" (UID: \"e86ef8d1-534c-4d8d-b07a-c5592c8311db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" Jul 16 12:18:04.656266 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.656235 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac970c7-8ff5-4d23-b93c-54d231fa2b08-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bac970c7-8ff5-4d23-b93c-54d231fa2b08" (UID: "bac970c7-8ff5-4d23-b93c-54d231fa2b08"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:18:04.656375 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.656353 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpvj5\" (UniqueName: \"kubernetes.io/projected/e86ef8d1-534c-4d8d-b07a-c5592c8311db-kube-api-access-gpvj5\") pod \"kuadrant-operator-controller-manager-86878c769f-jng5s\" (UID: \"e86ef8d1-534c-4d8d-b07a-c5592c8311db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" Jul 16 12:18:04.657035 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.656830 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bac970c7-8ff5-4d23-b93c-54d231fa2b08-extensions-socket-volume\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:18:04.658190 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.658164 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac970c7-8ff5-4d23-b93c-54d231fa2b08-kube-api-access-86d4q" (OuterVolumeSpecName: "kube-api-access-86d4q") pod "bac970c7-8ff5-4d23-b93c-54d231fa2b08" (UID: "bac970c7-8ff5-4d23-b93c-54d231fa2b08"). InnerVolumeSpecName "kube-api-access-86d4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:18:04.758130 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.758089 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e86ef8d1-534c-4d8d-b07a-c5592c8311db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jng5s\" (UID: \"e86ef8d1-534c-4d8d-b07a-c5592c8311db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" Jul 16 12:18:04.758130 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.758128 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpvj5\" (UniqueName: \"kubernetes.io/projected/e86ef8d1-534c-4d8d-b07a-c5592c8311db-kube-api-access-gpvj5\") pod \"kuadrant-operator-controller-manager-86878c769f-jng5s\" (UID: \"e86ef8d1-534c-4d8d-b07a-c5592c8311db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" Jul 16 12:18:04.758339 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.758230 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86d4q\" (UniqueName: \"kubernetes.io/projected/bac970c7-8ff5-4d23-b93c-54d231fa2b08-kube-api-access-86d4q\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:18:04.758493 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.758460 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e86ef8d1-534c-4d8d-b07a-c5592c8311db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jng5s\" (UID: \"e86ef8d1-534c-4d8d-b07a-c5592c8311db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" Jul 16 12:18:04.767819 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.767789 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpvj5\" (UniqueName: \"kubernetes.io/projected/e86ef8d1-534c-4d8d-b07a-c5592c8311db-kube-api-access-gpvj5\") pod \"kuadrant-operator-controller-manager-86878c769f-jng5s\" (UID: \"e86ef8d1-534c-4d8d-b07a-c5592c8311db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" Jul 16 12:18:04.934942 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:04.934844 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" Jul 16 12:18:05.070860 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:05.070834 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s"] Jul 16 12:18:05.073427 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:18:05.073393 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86ef8d1_534c_4d8d_b07a_c5592c8311db.slice/crio-b172a05146c43966be888c9c36e974537f4391035099fa0ad388c6ccd967bf8d WatchSource:0}: Error finding container b172a05146c43966be888c9c36e974537f4391035099fa0ad388c6ccd967bf8d: Status 404 returned error can't find the container with id b172a05146c43966be888c9c36e974537f4391035099fa0ad388c6ccd967bf8d Jul 16 12:18:05.109232 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:05.109192 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" event={"ID":"e86ef8d1-534c-4d8d-b07a-c5592c8311db","Type":"ContainerStarted","Data":"b172a05146c43966be888c9c36e974537f4391035099fa0ad388c6ccd967bf8d"} Jul 16 12:18:05.110545 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:05.110505 2559 generic.go:356] "Generic (PLEG): container finished" podID="bac970c7-8ff5-4d23-b93c-54d231fa2b08" containerID="d40a96d90b74ab43561a7823bed54fa6f41bbd95fa827bb414603adde01357f3" exitCode=0 Jul 16 12:18:05.110693 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:05.110541 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" event={"ID":"bac970c7-8ff5-4d23-b93c-54d231fa2b08","Type":"ContainerDied","Data":"d40a96d90b74ab43561a7823bed54fa6f41bbd95fa827bb414603adde01357f3"} Jul 16 12:18:05.110693 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:05.110559 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" Jul 16 12:18:05.110693 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:05.110594 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl" event={"ID":"bac970c7-8ff5-4d23-b93c-54d231fa2b08","Type":"ContainerDied","Data":"56051fe5e46da1361769fd5f6f7aa6fbe8f8cbf28d02b57fdaa1ef2ea14c2291"} Jul 16 12:18:05.110693 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:05.110620 2559 scope.go:117] "RemoveContainer" containerID="d40a96d90b74ab43561a7823bed54fa6f41bbd95fa827bb414603adde01357f3" Jul 16 12:18:05.119285 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:05.119269 2559 scope.go:117] "RemoveContainer" containerID="d40a96d90b74ab43561a7823bed54fa6f41bbd95fa827bb414603adde01357f3" Jul 16 12:18:05.119572 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:18:05.119551 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d40a96d90b74ab43561a7823bed54fa6f41bbd95fa827bb414603adde01357f3\": container with ID starting with d40a96d90b74ab43561a7823bed54fa6f41bbd95fa827bb414603adde01357f3 not found: ID does not exist" containerID="d40a96d90b74ab43561a7823bed54fa6f41bbd95fa827bb414603adde01357f3" Jul 16 12:18:05.119632 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:05.119581 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d40a96d90b74ab43561a7823bed54fa6f41bbd95fa827bb414603adde01357f3"} err="failed to get container status \"d40a96d90b74ab43561a7823bed54fa6f41bbd95fa827bb414603adde01357f3\": rpc error: code = NotFound desc = could not find container \"d40a96d90b74ab43561a7823bed54fa6f41bbd95fa827bb414603adde01357f3\": container with ID starting with d40a96d90b74ab43561a7823bed54fa6f41bbd95fa827bb414603adde01357f3 not found: ID does not exist" Jul 16 12:18:05.155116 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:05.155066 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl"] Jul 16 12:18:05.157637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:05.157612 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vb7nl"] Jul 16 12:18:06.116953 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:06.116915 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" event={"ID":"e86ef8d1-534c-4d8d-b07a-c5592c8311db","Type":"ContainerStarted","Data":"1c922ceabe279bb079b2170a7b3f4e2646deea38fcc3be8397bd5ac7678d7a03"} Jul 16 12:18:06.117380 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:06.117027 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" Jul 16 12:18:06.171579 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:06.171519 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" podStartSLOduration=2.171502442 podStartE2EDuration="2.171502442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:18:06.165572574 +0000 UTC m=+664.315280044" watchObservedRunningTime="2026-07-16 12:18:06.171502442 +0000 UTC m=+664.321209876" Jul 16 12:18:06.556494 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:06.556460 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bac970c7-8ff5-4d23-b93c-54d231fa2b08" path="/var/lib/kubelet/pods/bac970c7-8ff5-4d23-b93c-54d231fa2b08/volumes" Jul 16 12:18:17.125331 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:17.125299 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" Jul 16 12:18:24.642449 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:24.642410 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-crr5k"] Jul 16 12:18:24.646260 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:24.646235 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-crr5k" Jul 16 12:18:24.649228 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:24.649207 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-r2mth\"" Jul 16 12:18:24.659632 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:24.659605 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-crr5k"] Jul 16 12:18:24.741092 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:24.741055 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdrp5\" (UniqueName: \"kubernetes.io/projected/1abf668d-ecae-4b10-b14e-278230ad26e0-kube-api-access-rdrp5\") pod \"authorino-58cd955675-crr5k\" (UID: \"1abf668d-ecae-4b10-b14e-278230ad26e0\") " pod="kuadrant-system/authorino-58cd955675-crr5k" Jul 16 12:18:24.841750 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:24.841703 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdrp5\" (UniqueName: \"kubernetes.io/projected/1abf668d-ecae-4b10-b14e-278230ad26e0-kube-api-access-rdrp5\") pod \"authorino-58cd955675-crr5k\" (UID: \"1abf668d-ecae-4b10-b14e-278230ad26e0\") " pod="kuadrant-system/authorino-58cd955675-crr5k" Jul 16 12:18:24.851987 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:24.851960 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdrp5\" (UniqueName: \"kubernetes.io/projected/1abf668d-ecae-4b10-b14e-278230ad26e0-kube-api-access-rdrp5\") pod \"authorino-58cd955675-crr5k\" (UID: \"1abf668d-ecae-4b10-b14e-278230ad26e0\") " pod="kuadrant-system/authorino-58cd955675-crr5k" Jul 16 12:18:24.855835 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:24.855810 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-wvkjv"] Jul 16 12:18:24.859183 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:24.859162 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wvkjv" Jul 16 12:18:24.879897 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:24.879865 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wvkjv"] Jul 16 12:18:24.942161 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:24.942080 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4h24\" (UniqueName: \"kubernetes.io/projected/54b61ca7-a449-4691-8f99-e9410ccda61c-kube-api-access-w4h24\") pod \"authorino-75d556bd77-wvkjv\" (UID: \"54b61ca7-a449-4691-8f99-e9410ccda61c\") " pod="kuadrant-system/authorino-75d556bd77-wvkjv" Jul 16 12:18:24.955998 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:24.955972 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-crr5k" Jul 16 12:18:25.043301 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:25.043273 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4h24\" (UniqueName: \"kubernetes.io/projected/54b61ca7-a449-4691-8f99-e9410ccda61c-kube-api-access-w4h24\") pod \"authorino-75d556bd77-wvkjv\" (UID: \"54b61ca7-a449-4691-8f99-e9410ccda61c\") " pod="kuadrant-system/authorino-75d556bd77-wvkjv" Jul 16 12:18:25.052929 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:25.052897 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4h24\" (UniqueName: \"kubernetes.io/projected/54b61ca7-a449-4691-8f99-e9410ccda61c-kube-api-access-w4h24\") pod \"authorino-75d556bd77-wvkjv\" (UID: \"54b61ca7-a449-4691-8f99-e9410ccda61c\") " pod="kuadrant-system/authorino-75d556bd77-wvkjv" Jul 16 12:18:25.096034 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:25.096004 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-crr5k"] Jul 16 12:18:25.098799 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:18:25.098772 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1abf668d_ecae_4b10_b14e_278230ad26e0.slice/crio-90ec16f855be48b0936dfdc05ede571dd46cf14da942954cbb2592d2eb95984f WatchSource:0}: Error finding container 90ec16f855be48b0936dfdc05ede571dd46cf14da942954cbb2592d2eb95984f: Status 404 returned error can't find the container with id 90ec16f855be48b0936dfdc05ede571dd46cf14da942954cbb2592d2eb95984f Jul 16 12:18:25.168552 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:25.168521 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wvkjv" Jul 16 12:18:25.208145 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:25.208054 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-crr5k" event={"ID":"1abf668d-ecae-4b10-b14e-278230ad26e0","Type":"ContainerStarted","Data":"90ec16f855be48b0936dfdc05ede571dd46cf14da942954cbb2592d2eb95984f"} Jul 16 12:18:25.309053 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:25.309021 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wvkjv"] Jul 16 12:18:25.311913 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:18:25.311883 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54b61ca7_a449_4691_8f99_e9410ccda61c.slice/crio-02c237351cb9d9735b36e8fbec4cceb82b2cefa9eb9a28ded6784e1fb94fea5b WatchSource:0}: Error finding container 02c237351cb9d9735b36e8fbec4cceb82b2cefa9eb9a28ded6784e1fb94fea5b: Status 404 returned error can't find the container with id 02c237351cb9d9735b36e8fbec4cceb82b2cefa9eb9a28ded6784e1fb94fea5b Jul 16 12:18:26.217502 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:26.217441 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wvkjv" event={"ID":"54b61ca7-a449-4691-8f99-e9410ccda61c","Type":"ContainerStarted","Data":"02c237351cb9d9735b36e8fbec4cceb82b2cefa9eb9a28ded6784e1fb94fea5b"} Jul 16 12:18:29.237955 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:29.237915 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-crr5k" event={"ID":"1abf668d-ecae-4b10-b14e-278230ad26e0","Type":"ContainerStarted","Data":"d6fd46144384f3287aa6bbdbf8876393d342e135b1192150da804b82898c3f7d"} Jul 16 12:18:29.239435 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:29.239408 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wvkjv" event={"ID":"54b61ca7-a449-4691-8f99-e9410ccda61c","Type":"ContainerStarted","Data":"32698c565b1be8ab3c1e18f0ae55e2130df408c62eb471f076d99dc4f06cc535"} Jul 16 12:18:29.257431 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:29.257374 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-crr5k" podStartSLOduration=1.730816381 podStartE2EDuration="5.257358427s" totalImagesPullingTime="3.526542046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:18:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:18:29.256210822 +0000 UTC m=+687.405918250" watchObservedRunningTime="2026-07-16 12:18:29.257358427 +0000 UTC m=+687.407065860" Jul 16 12:18:29.322995 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:29.322937 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-wvkjv" podStartSLOduration=1.998565944 podStartE2EDuration="5.32291771s" totalImagesPullingTime="3.324351766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:18:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:18:29.280580749 +0000 UTC m=+687.430288181" watchObservedRunningTime="2026-07-16 12:18:29.32291771 +0000 UTC m=+687.472625142" Jul 16 12:18:29.323983 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:29.323966 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-crr5k"] Jul 16 12:18:31.248219 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:31.248177 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-crr5k" podUID="1abf668d-ecae-4b10-b14e-278230ad26e0" containerName="authorino" containerID="cri-o://d6fd46144384f3287aa6bbdbf8876393d342e135b1192150da804b82898c3f7d" gracePeriod=30 Jul 16 12:18:31.500084 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:31.500023 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-crr5k" Jul 16 12:18:31.607071 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:31.607038 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdrp5\" (UniqueName: \"kubernetes.io/projected/1abf668d-ecae-4b10-b14e-278230ad26e0-kube-api-access-rdrp5\") pod \"1abf668d-ecae-4b10-b14e-278230ad26e0\" (UID: \"1abf668d-ecae-4b10-b14e-278230ad26e0\") " Jul 16 12:18:31.609085 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:31.609053 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1abf668d-ecae-4b10-b14e-278230ad26e0-kube-api-access-rdrp5" (OuterVolumeSpecName: "kube-api-access-rdrp5") pod "1abf668d-ecae-4b10-b14e-278230ad26e0" (UID: "1abf668d-ecae-4b10-b14e-278230ad26e0"). InnerVolumeSpecName "kube-api-access-rdrp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:18:31.708707 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:31.708652 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdrp5\" (UniqueName: \"kubernetes.io/projected/1abf668d-ecae-4b10-b14e-278230ad26e0-kube-api-access-rdrp5\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:18:32.254377 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:32.254343 2559 generic.go:356] "Generic (PLEG): container finished" podID="1abf668d-ecae-4b10-b14e-278230ad26e0" containerID="d6fd46144384f3287aa6bbdbf8876393d342e135b1192150da804b82898c3f7d" exitCode=0 Jul 16 12:18:32.254883 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:32.254390 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-crr5k" Jul 16 12:18:32.254883 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:32.254435 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-crr5k" event={"ID":"1abf668d-ecae-4b10-b14e-278230ad26e0","Type":"ContainerDied","Data":"d6fd46144384f3287aa6bbdbf8876393d342e135b1192150da804b82898c3f7d"} Jul 16 12:18:32.254883 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:32.254481 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-crr5k" event={"ID":"1abf668d-ecae-4b10-b14e-278230ad26e0","Type":"ContainerDied","Data":"90ec16f855be48b0936dfdc05ede571dd46cf14da942954cbb2592d2eb95984f"} Jul 16 12:18:32.254883 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:32.254501 2559 scope.go:117] "RemoveContainer" containerID="d6fd46144384f3287aa6bbdbf8876393d342e135b1192150da804b82898c3f7d" Jul 16 12:18:32.262805 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:32.262789 2559 scope.go:117] "RemoveContainer" containerID="d6fd46144384f3287aa6bbdbf8876393d342e135b1192150da804b82898c3f7d" Jul 16 12:18:32.263069 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:18:32.263051 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6fd46144384f3287aa6bbdbf8876393d342e135b1192150da804b82898c3f7d\": container with ID starting with d6fd46144384f3287aa6bbdbf8876393d342e135b1192150da804b82898c3f7d not found: ID does not exist" containerID="d6fd46144384f3287aa6bbdbf8876393d342e135b1192150da804b82898c3f7d" Jul 16 12:18:32.263113 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:32.263079 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6fd46144384f3287aa6bbdbf8876393d342e135b1192150da804b82898c3f7d"} err="failed to get container status \"d6fd46144384f3287aa6bbdbf8876393d342e135b1192150da804b82898c3f7d\": rpc error: code = NotFound desc = could not find container \"d6fd46144384f3287aa6bbdbf8876393d342e135b1192150da804b82898c3f7d\": container with ID starting with d6fd46144384f3287aa6bbdbf8876393d342e135b1192150da804b82898c3f7d not found: ID does not exist" Jul 16 12:18:32.283394 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:32.283367 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-crr5k"] Jul 16 12:18:32.288198 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:32.288168 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-crr5k"] Jul 16 12:18:32.558063 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:32.557969 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1abf668d-ecae-4b10-b14e-278230ad26e0" path="/var/lib/kubelet/pods/1abf668d-ecae-4b10-b14e-278230ad26e0/volumes" Jul 16 12:18:58.251831 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.251732 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-v8hpm"] Jul 16 12:18:58.252495 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.252113 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1abf668d-ecae-4b10-b14e-278230ad26e0" containerName="authorino" Jul 16 12:18:58.252495 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.252127 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1abf668d-ecae-4b10-b14e-278230ad26e0" containerName="authorino" Jul 16 12:18:58.252495 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.252201 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1abf668d-ecae-4b10-b14e-278230ad26e0" containerName="authorino" Jul 16 12:18:58.255016 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.255000 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-v8hpm" Jul 16 12:18:58.262974 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.262084 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-v8hpm"] Jul 16 12:18:58.339990 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.339954 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhqrw\" (UniqueName: \"kubernetes.io/projected/e63a2a39-1aea-409c-bb9e-7c618f01b29a-kube-api-access-zhqrw\") pod \"authorino-5d4b67c446-v8hpm\" (UID: \"e63a2a39-1aea-409c-bb9e-7c618f01b29a\") " pod="kuadrant-system/authorino-5d4b67c446-v8hpm" Jul 16 12:18:58.441212 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.441167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhqrw\" (UniqueName: \"kubernetes.io/projected/e63a2a39-1aea-409c-bb9e-7c618f01b29a-kube-api-access-zhqrw\") pod \"authorino-5d4b67c446-v8hpm\" (UID: \"e63a2a39-1aea-409c-bb9e-7c618f01b29a\") " pod="kuadrant-system/authorino-5d4b67c446-v8hpm" Jul 16 12:18:58.450098 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.450068 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhqrw\" (UniqueName: \"kubernetes.io/projected/e63a2a39-1aea-409c-bb9e-7c618f01b29a-kube-api-access-zhqrw\") pod \"authorino-5d4b67c446-v8hpm\" (UID: \"e63a2a39-1aea-409c-bb9e-7c618f01b29a\") " pod="kuadrant-system/authorino-5d4b67c446-v8hpm" Jul 16 12:18:58.522997 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.522906 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-v8hpm"] Jul 16 12:18:58.523181 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.523169 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-v8hpm" Jul 16 12:18:58.586360 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.586320 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d8f5dd56-lvvq5"] Jul 16 12:18:58.590479 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.590445 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d8f5dd56-lvvq5" Jul 16 12:18:58.602816 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.602786 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d8f5dd56-lvvq5"] Jul 16 12:18:58.642966 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.642928 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c884p\" (UniqueName: \"kubernetes.io/projected/6b719e5f-72cc-4f63-baff-1e253cd5e640-kube-api-access-c884p\") pod \"authorino-7d8f5dd56-lvvq5\" (UID: \"6b719e5f-72cc-4f63-baff-1e253cd5e640\") " pod="kuadrant-system/authorino-7d8f5dd56-lvvq5" Jul 16 12:18:58.674202 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.674156 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-v8hpm"] Jul 16 12:18:58.679168 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:18:58.679140 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode63a2a39_1aea_409c_bb9e_7c618f01b29a.slice/crio-8d694647ea38d2d04bca21d985757e34f4c05aab729328f19d873fd7bdddf39b WatchSource:0}: Error finding container 8d694647ea38d2d04bca21d985757e34f4c05aab729328f19d873fd7bdddf39b: Status 404 returned error can't find the container with id 8d694647ea38d2d04bca21d985757e34f4c05aab729328f19d873fd7bdddf39b Jul 16 12:18:58.744024 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.743982 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c884p\" (UniqueName: \"kubernetes.io/projected/6b719e5f-72cc-4f63-baff-1e253cd5e640-kube-api-access-c884p\") pod \"authorino-7d8f5dd56-lvvq5\" (UID: \"6b719e5f-72cc-4f63-baff-1e253cd5e640\") " pod="kuadrant-system/authorino-7d8f5dd56-lvvq5" Jul 16 12:18:58.762469 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.762436 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c884p\" (UniqueName: \"kubernetes.io/projected/6b719e5f-72cc-4f63-baff-1e253cd5e640-kube-api-access-c884p\") pod \"authorino-7d8f5dd56-lvvq5\" (UID: \"6b719e5f-72cc-4f63-baff-1e253cd5e640\") " pod="kuadrant-system/authorino-7d8f5dd56-lvvq5" Jul 16 12:18:58.875055 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.874973 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d8f5dd56-lvvq5"] Jul 16 12:18:58.875256 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.875244 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d8f5dd56-lvvq5" Jul 16 12:18:58.903509 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.903475 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c765588b4-d6jxz"] Jul 16 12:18:58.908617 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.908590 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c765588b4-d6jxz" Jul 16 12:18:58.911341 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.911315 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 16 12:18:58.918804 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:58.918776 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c765588b4-d6jxz"] Jul 16 12:18:59.035721 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.035696 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d8f5dd56-lvvq5"] Jul 16 12:18:59.038174 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:18:59.038148 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b719e5f_72cc_4f63_baff_1e253cd5e640.slice/crio-59808243b98ada0edecc70ab80f6347403775581f6a48ab1c987b6f2161eb464 WatchSource:0}: Error finding container 59808243b98ada0edecc70ab80f6347403775581f6a48ab1c987b6f2161eb464: Status 404 returned error can't find the container with id 59808243b98ada0edecc70ab80f6347403775581f6a48ab1c987b6f2161eb464 Jul 16 12:18:59.046775 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.046748 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fcea7d11-59e9-454c-99e0-5dee5da3823e-tls-cert\") pod \"authorino-5c765588b4-d6jxz\" (UID: \"fcea7d11-59e9-454c-99e0-5dee5da3823e\") " pod="kuadrant-system/authorino-5c765588b4-d6jxz" Jul 16 12:18:59.046888 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.046815 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s24jq\" (UniqueName: \"kubernetes.io/projected/fcea7d11-59e9-454c-99e0-5dee5da3823e-kube-api-access-s24jq\") pod \"authorino-5c765588b4-d6jxz\" (UID: \"fcea7d11-59e9-454c-99e0-5dee5da3823e\") " pod="kuadrant-system/authorino-5c765588b4-d6jxz" Jul 16 12:18:59.147855 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.147756 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fcea7d11-59e9-454c-99e0-5dee5da3823e-tls-cert\") pod \"authorino-5c765588b4-d6jxz\" (UID: \"fcea7d11-59e9-454c-99e0-5dee5da3823e\") " pod="kuadrant-system/authorino-5c765588b4-d6jxz" Jul 16 12:18:59.147855 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.147849 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s24jq\" (UniqueName: \"kubernetes.io/projected/fcea7d11-59e9-454c-99e0-5dee5da3823e-kube-api-access-s24jq\") pod \"authorino-5c765588b4-d6jxz\" (UID: \"fcea7d11-59e9-454c-99e0-5dee5da3823e\") " pod="kuadrant-system/authorino-5c765588b4-d6jxz" Jul 16 12:18:59.150207 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.150186 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fcea7d11-59e9-454c-99e0-5dee5da3823e-tls-cert\") pod \"authorino-5c765588b4-d6jxz\" (UID: \"fcea7d11-59e9-454c-99e0-5dee5da3823e\") " pod="kuadrant-system/authorino-5c765588b4-d6jxz" Jul 16 12:18:59.156471 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.156445 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s24jq\" (UniqueName: \"kubernetes.io/projected/fcea7d11-59e9-454c-99e0-5dee5da3823e-kube-api-access-s24jq\") pod \"authorino-5c765588b4-d6jxz\" (UID: \"fcea7d11-59e9-454c-99e0-5dee5da3823e\") " pod="kuadrant-system/authorino-5c765588b4-d6jxz" Jul 16 12:18:59.219787 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.219747 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c765588b4-d6jxz" Jul 16 12:18:59.378943 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.378916 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c765588b4-d6jxz"] Jul 16 12:18:59.381183 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:18:59.381151 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcea7d11_59e9_454c_99e0_5dee5da3823e.slice/crio-9d1f5025b04373978fdeb7384524d1fc2fa397e6beae1f3f5e695c33b81201cd WatchSource:0}: Error finding container 9d1f5025b04373978fdeb7384524d1fc2fa397e6beae1f3f5e695c33b81201cd: Status 404 returned error can't find the container with id 9d1f5025b04373978fdeb7384524d1fc2fa397e6beae1f3f5e695c33b81201cd Jul 16 12:18:59.386358 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.386327 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-v8hpm" podUID="e63a2a39-1aea-409c-bb9e-7c618f01b29a" containerName="authorino" containerID="cri-o://925346a35d2a915d58e687005d7c8e5faaa07e4bc1149834308635ace6b2e610" gracePeriod=30 Jul 16 12:18:59.386509 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.386326 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-v8hpm" event={"ID":"e63a2a39-1aea-409c-bb9e-7c618f01b29a","Type":"ContainerStarted","Data":"925346a35d2a915d58e687005d7c8e5faaa07e4bc1149834308635ace6b2e610"} Jul 16 12:18:59.386633 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.386543 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-v8hpm" event={"ID":"e63a2a39-1aea-409c-bb9e-7c618f01b29a","Type":"ContainerStarted","Data":"8d694647ea38d2d04bca21d985757e34f4c05aab729328f19d873fd7bdddf39b"} Jul 16 12:18:59.387648 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.387623 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d8f5dd56-lvvq5" event={"ID":"6b719e5f-72cc-4f63-baff-1e253cd5e640","Type":"ContainerStarted","Data":"59808243b98ada0edecc70ab80f6347403775581f6a48ab1c987b6f2161eb464"} Jul 16 12:18:59.402869 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.402780 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-v8hpm" podStartSLOduration=1.090585153 podStartE2EDuration="1.402762133s" totalImagesPullingTime="312.17698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:18:59.402618157 +0000 UTC m=+717.552325590" watchObservedRunningTime="2026-07-16 12:18:59.402762133 +0000 UTC m=+717.552469568" Jul 16 12:18:59.699477 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.699443 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-v8hpm" Jul 16 12:18:59.752622 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.752588 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhqrw\" (UniqueName: \"kubernetes.io/projected/e63a2a39-1aea-409c-bb9e-7c618f01b29a-kube-api-access-zhqrw\") pod \"e63a2a39-1aea-409c-bb9e-7c618f01b29a\" (UID: \"e63a2a39-1aea-409c-bb9e-7c618f01b29a\") " Jul 16 12:18:59.754669 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.754636 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63a2a39-1aea-409c-bb9e-7c618f01b29a-kube-api-access-zhqrw" (OuterVolumeSpecName: "kube-api-access-zhqrw") pod "e63a2a39-1aea-409c-bb9e-7c618f01b29a" (UID: "e63a2a39-1aea-409c-bb9e-7c618f01b29a"). InnerVolumeSpecName "kube-api-access-zhqrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:18:59.853622 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:18:59.853578 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhqrw\" (UniqueName: \"kubernetes.io/projected/e63a2a39-1aea-409c-bb9e-7c618f01b29a-kube-api-access-zhqrw\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:19:00.393719 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.393661 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d8f5dd56-lvvq5" event={"ID":"6b719e5f-72cc-4f63-baff-1e253cd5e640","Type":"ContainerStarted","Data":"739e48d21f2f85e1e0d3b2b2d581a71b825a0238c2d7b87c63bd1ba76cffd47f"} Jul 16 12:19:00.393719 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.393671 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7d8f5dd56-lvvq5" podUID="6b719e5f-72cc-4f63-baff-1e253cd5e640" containerName="authorino" containerID="cri-o://739e48d21f2f85e1e0d3b2b2d581a71b825a0238c2d7b87c63bd1ba76cffd47f" gracePeriod=30 Jul 16 12:19:00.395313 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.395283 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c765588b4-d6jxz" event={"ID":"fcea7d11-59e9-454c-99e0-5dee5da3823e","Type":"ContainerStarted","Data":"9a085a2359c445d2bc74d46fa5712daec7c72f67981007fb370867aafc5cc9b4"} Jul 16 12:19:00.395448 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.395325 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c765588b4-d6jxz" event={"ID":"fcea7d11-59e9-454c-99e0-5dee5da3823e","Type":"ContainerStarted","Data":"9d1f5025b04373978fdeb7384524d1fc2fa397e6beae1f3f5e695c33b81201cd"} Jul 16 12:19:00.396607 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.396577 2559 generic.go:356] "Generic (PLEG): container finished" podID="e63a2a39-1aea-409c-bb9e-7c618f01b29a" containerID="925346a35d2a915d58e687005d7c8e5faaa07e4bc1149834308635ace6b2e610" exitCode=0 Jul 16 12:19:00.396754 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.396622 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-v8hpm" Jul 16 12:19:00.396754 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.396630 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-v8hpm" event={"ID":"e63a2a39-1aea-409c-bb9e-7c618f01b29a","Type":"ContainerDied","Data":"925346a35d2a915d58e687005d7c8e5faaa07e4bc1149834308635ace6b2e610"} Jul 16 12:19:00.396754 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.396733 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-v8hpm" event={"ID":"e63a2a39-1aea-409c-bb9e-7c618f01b29a","Type":"ContainerDied","Data":"8d694647ea38d2d04bca21d985757e34f4c05aab729328f19d873fd7bdddf39b"} Jul 16 12:19:00.396905 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.396760 2559 scope.go:117] "RemoveContainer" containerID="925346a35d2a915d58e687005d7c8e5faaa07e4bc1149834308635ace6b2e610" Jul 16 12:19:00.405894 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.405874 2559 scope.go:117] "RemoveContainer" containerID="925346a35d2a915d58e687005d7c8e5faaa07e4bc1149834308635ace6b2e610" Jul 16 12:19:00.406348 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:19:00.406318 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925346a35d2a915d58e687005d7c8e5faaa07e4bc1149834308635ace6b2e610\": container with ID starting with 925346a35d2a915d58e687005d7c8e5faaa07e4bc1149834308635ace6b2e610 not found: ID does not exist" containerID="925346a35d2a915d58e687005d7c8e5faaa07e4bc1149834308635ace6b2e610" Jul 16 12:19:00.406439 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.406359 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925346a35d2a915d58e687005d7c8e5faaa07e4bc1149834308635ace6b2e610"} err="failed to get container status \"925346a35d2a915d58e687005d7c8e5faaa07e4bc1149834308635ace6b2e610\": rpc error: code = NotFound desc = could not find container \"925346a35d2a915d58e687005d7c8e5faaa07e4bc1149834308635ace6b2e610\": container with ID starting with 925346a35d2a915d58e687005d7c8e5faaa07e4bc1149834308635ace6b2e610 not found: ID does not exist" Jul 16 12:19:00.425755 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.425702 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7d8f5dd56-lvvq5" podStartSLOduration=1.859843001 podStartE2EDuration="2.42566735s" totalImagesPullingTime="565.824349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:19:00.424046662 +0000 UTC m=+718.573754098" watchObservedRunningTime="2026-07-16 12:19:00.42566735 +0000 UTC m=+718.575374783" Jul 16 12:19:00.449872 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.449819 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5c765588b4-d6jxz" podStartSLOduration=1.901285026 podStartE2EDuration="2.449804123s" totalImagesPullingTime="548.519097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:19:00.448707871 +0000 UTC m=+718.598415306" watchObservedRunningTime="2026-07-16 12:19:00.449804123 +0000 UTC m=+718.599511556" Jul 16 12:19:00.473308 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.473273 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-v8hpm"] Jul 16 12:19:00.477791 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.477761 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-v8hpm"] Jul 16 12:19:00.483888 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.483865 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wvkjv"] Jul 16 12:19:00.484059 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.484035 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-wvkjv" podUID="54b61ca7-a449-4691-8f99-e9410ccda61c" containerName="authorino" containerID="cri-o://32698c565b1be8ab3c1e18f0ae55e2130df408c62eb471f076d99dc4f06cc535" gracePeriod=30 Jul 16 12:19:00.557813 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.557781 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e63a2a39-1aea-409c-bb9e-7c618f01b29a" path="/var/lib/kubelet/pods/e63a2a39-1aea-409c-bb9e-7c618f01b29a/volumes" Jul 16 12:19:00.755917 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.755890 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d8f5dd56-lvvq5" Jul 16 12:19:00.758900 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.758878 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wvkjv" Jul 16 12:19:00.862769 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.862735 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c884p\" (UniqueName: \"kubernetes.io/projected/6b719e5f-72cc-4f63-baff-1e253cd5e640-kube-api-access-c884p\") pod \"6b719e5f-72cc-4f63-baff-1e253cd5e640\" (UID: \"6b719e5f-72cc-4f63-baff-1e253cd5e640\") " Jul 16 12:19:00.862950 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.862789 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4h24\" (UniqueName: \"kubernetes.io/projected/54b61ca7-a449-4691-8f99-e9410ccda61c-kube-api-access-w4h24\") pod \"54b61ca7-a449-4691-8f99-e9410ccda61c\" (UID: \"54b61ca7-a449-4691-8f99-e9410ccda61c\") " Jul 16 12:19:00.864901 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.864876 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b719e5f-72cc-4f63-baff-1e253cd5e640-kube-api-access-c884p" (OuterVolumeSpecName: "kube-api-access-c884p") pod "6b719e5f-72cc-4f63-baff-1e253cd5e640" (UID: "6b719e5f-72cc-4f63-baff-1e253cd5e640"). InnerVolumeSpecName "kube-api-access-c884p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:19:00.864990 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.864910 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b61ca7-a449-4691-8f99-e9410ccda61c-kube-api-access-w4h24" (OuterVolumeSpecName: "kube-api-access-w4h24") pod "54b61ca7-a449-4691-8f99-e9410ccda61c" (UID: "54b61ca7-a449-4691-8f99-e9410ccda61c"). InnerVolumeSpecName "kube-api-access-w4h24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:19:00.963803 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.963718 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c884p\" (UniqueName: \"kubernetes.io/projected/6b719e5f-72cc-4f63-baff-1e253cd5e640-kube-api-access-c884p\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:19:00.963803 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:00.963750 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4h24\" (UniqueName: \"kubernetes.io/projected/54b61ca7-a449-4691-8f99-e9410ccda61c-kube-api-access-w4h24\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:19:01.402882 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.402841 2559 generic.go:356] "Generic (PLEG): container finished" podID="54b61ca7-a449-4691-8f99-e9410ccda61c" containerID="32698c565b1be8ab3c1e18f0ae55e2130df408c62eb471f076d99dc4f06cc535" exitCode=0 Jul 16 12:19:01.402882 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.402874 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wvkjv" Jul 16 12:19:01.403373 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.402914 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wvkjv" event={"ID":"54b61ca7-a449-4691-8f99-e9410ccda61c","Type":"ContainerDied","Data":"32698c565b1be8ab3c1e18f0ae55e2130df408c62eb471f076d99dc4f06cc535"} Jul 16 12:19:01.403373 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.402950 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wvkjv" event={"ID":"54b61ca7-a449-4691-8f99-e9410ccda61c","Type":"ContainerDied","Data":"02c237351cb9d9735b36e8fbec4cceb82b2cefa9eb9a28ded6784e1fb94fea5b"} Jul 16 12:19:01.403373 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.402969 2559 scope.go:117] "RemoveContainer" containerID="32698c565b1be8ab3c1e18f0ae55e2130df408c62eb471f076d99dc4f06cc535" Jul 16 12:19:01.404351 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.404326 2559 generic.go:356] "Generic (PLEG): container finished" podID="6b719e5f-72cc-4f63-baff-1e253cd5e640" containerID="739e48d21f2f85e1e0d3b2b2d581a71b825a0238c2d7b87c63bd1ba76cffd47f" exitCode=0 Jul 16 12:19:01.404468 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.404361 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d8f5dd56-lvvq5" Jul 16 12:19:01.404468 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.404406 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d8f5dd56-lvvq5" event={"ID":"6b719e5f-72cc-4f63-baff-1e253cd5e640","Type":"ContainerDied","Data":"739e48d21f2f85e1e0d3b2b2d581a71b825a0238c2d7b87c63bd1ba76cffd47f"} Jul 16 12:19:01.404468 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.404436 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d8f5dd56-lvvq5" event={"ID":"6b719e5f-72cc-4f63-baff-1e253cd5e640","Type":"ContainerDied","Data":"59808243b98ada0edecc70ab80f6347403775581f6a48ab1c987b6f2161eb464"} Jul 16 12:19:01.412721 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.412699 2559 scope.go:117] "RemoveContainer" containerID="32698c565b1be8ab3c1e18f0ae55e2130df408c62eb471f076d99dc4f06cc535" Jul 16 12:19:01.412989 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:19:01.412967 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32698c565b1be8ab3c1e18f0ae55e2130df408c62eb471f076d99dc4f06cc535\": container with ID starting with 32698c565b1be8ab3c1e18f0ae55e2130df408c62eb471f076d99dc4f06cc535 not found: ID does not exist" containerID="32698c565b1be8ab3c1e18f0ae55e2130df408c62eb471f076d99dc4f06cc535" Jul 16 12:19:01.413049 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.413002 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32698c565b1be8ab3c1e18f0ae55e2130df408c62eb471f076d99dc4f06cc535"} err="failed to get container status \"32698c565b1be8ab3c1e18f0ae55e2130df408c62eb471f076d99dc4f06cc535\": rpc error: code = NotFound desc = could not find container \"32698c565b1be8ab3c1e18f0ae55e2130df408c62eb471f076d99dc4f06cc535\": container with ID starting with 32698c565b1be8ab3c1e18f0ae55e2130df408c62eb471f076d99dc4f06cc535 not found: ID does not exist" Jul 16 12:19:01.413049 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.413022 2559 scope.go:117] "RemoveContainer" containerID="739e48d21f2f85e1e0d3b2b2d581a71b825a0238c2d7b87c63bd1ba76cffd47f" Jul 16 12:19:01.421025 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.421007 2559 scope.go:117] "RemoveContainer" containerID="739e48d21f2f85e1e0d3b2b2d581a71b825a0238c2d7b87c63bd1ba76cffd47f" Jul 16 12:19:01.421289 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:19:01.421267 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"739e48d21f2f85e1e0d3b2b2d581a71b825a0238c2d7b87c63bd1ba76cffd47f\": container with ID starting with 739e48d21f2f85e1e0d3b2b2d581a71b825a0238c2d7b87c63bd1ba76cffd47f not found: ID does not exist" containerID="739e48d21f2f85e1e0d3b2b2d581a71b825a0238c2d7b87c63bd1ba76cffd47f" Jul 16 12:19:01.421361 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.421295 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"739e48d21f2f85e1e0d3b2b2d581a71b825a0238c2d7b87c63bd1ba76cffd47f"} err="failed to get container status \"739e48d21f2f85e1e0d3b2b2d581a71b825a0238c2d7b87c63bd1ba76cffd47f\": rpc error: code = NotFound desc = could not find container \"739e48d21f2f85e1e0d3b2b2d581a71b825a0238c2d7b87c63bd1ba76cffd47f\": container with ID starting with 739e48d21f2f85e1e0d3b2b2d581a71b825a0238c2d7b87c63bd1ba76cffd47f not found: ID does not exist" Jul 16 12:19:01.429601 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.429572 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d8f5dd56-lvvq5"] Jul 16 12:19:01.434249 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.434226 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7d8f5dd56-lvvq5"] Jul 16 12:19:01.449654 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.449623 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wvkjv"] Jul 16 12:19:01.453890 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:01.453863 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wvkjv"] Jul 16 12:19:02.556429 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:02.556388 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54b61ca7-a449-4691-8f99-e9410ccda61c" path="/var/lib/kubelet/pods/54b61ca7-a449-4691-8f99-e9410ccda61c/volumes" Jul 16 12:19:02.556807 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:02.556770 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b719e5f-72cc-4f63-baff-1e253cd5e640" path="/var/lib/kubelet/pods/6b719e5f-72cc-4f63-baff-1e253cd5e640/volumes" Jul 16 12:19:40.617086 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.617050 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-mbxwh"] Jul 16 12:19:40.617545 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.617378 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e63a2a39-1aea-409c-bb9e-7c618f01b29a" containerName="authorino" Jul 16 12:19:40.617545 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.617389 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e63a2a39-1aea-409c-bb9e-7c618f01b29a" containerName="authorino" Jul 16 12:19:40.617545 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.617402 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54b61ca7-a449-4691-8f99-e9410ccda61c" containerName="authorino" Jul 16 12:19:40.617545 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.617407 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="54b61ca7-a449-4691-8f99-e9410ccda61c" containerName="authorino" Jul 16 12:19:40.617545 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.617416 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b719e5f-72cc-4f63-baff-1e253cd5e640" containerName="authorino" Jul 16 12:19:40.617545 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.617421 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b719e5f-72cc-4f63-baff-1e253cd5e640" containerName="authorino" Jul 16 12:19:40.617545 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.617482 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b719e5f-72cc-4f63-baff-1e253cd5e640" containerName="authorino" Jul 16 12:19:40.617545 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.617489 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e63a2a39-1aea-409c-bb9e-7c618f01b29a" containerName="authorino" Jul 16 12:19:40.617545 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.617495 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="54b61ca7-a449-4691-8f99-e9410ccda61c" containerName="authorino" Jul 16 12:19:40.620811 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.620786 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-mbxwh" Jul 16 12:19:40.623558 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.623534 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-dc2jd\"" Jul 16 12:19:40.623700 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.623556 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 16 12:19:40.633194 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.633169 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-mbxwh"] Jul 16 12:19:40.704250 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.704209 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08da884b-cd39-43ed-bab1-62c7dfce73e8-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-mbxwh\" (UID: \"08da884b-cd39-43ed-bab1-62c7dfce73e8\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-mbxwh" Jul 16 12:19:40.704250 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.704258 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmxxr\" (UniqueName: \"kubernetes.io/projected/08da884b-cd39-43ed-bab1-62c7dfce73e8-kube-api-access-fmxxr\") pod \"payload-pre-processing-7677b774bb-mbxwh\" (UID: \"08da884b-cd39-43ed-bab1-62c7dfce73e8\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-mbxwh" Jul 16 12:19:40.805217 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.805178 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmxxr\" (UniqueName: \"kubernetes.io/projected/08da884b-cd39-43ed-bab1-62c7dfce73e8-kube-api-access-fmxxr\") pod \"payload-pre-processing-7677b774bb-mbxwh\" (UID: \"08da884b-cd39-43ed-bab1-62c7dfce73e8\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-mbxwh" Jul 16 12:19:40.805380 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.805271 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08da884b-cd39-43ed-bab1-62c7dfce73e8-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-mbxwh\" (UID: \"08da884b-cd39-43ed-bab1-62c7dfce73e8\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-mbxwh" Jul 16 12:19:40.805940 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.805921 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08da884b-cd39-43ed-bab1-62c7dfce73e8-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-mbxwh\" (UID: \"08da884b-cd39-43ed-bab1-62c7dfce73e8\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-mbxwh" Jul 16 12:19:40.814241 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.814214 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmxxr\" (UniqueName: \"kubernetes.io/projected/08da884b-cd39-43ed-bab1-62c7dfce73e8-kube-api-access-fmxxr\") pod \"payload-pre-processing-7677b774bb-mbxwh\" (UID: \"08da884b-cd39-43ed-bab1-62c7dfce73e8\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-mbxwh" Jul 16 12:19:40.931747 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:40.931645 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-mbxwh" Jul 16 12:19:41.093670 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:41.093631 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-mbxwh"] Jul 16 12:19:41.095942 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:19:41.095914 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08da884b_cd39_43ed_bab1_62c7dfce73e8.slice/crio-b6a286b7a3c84e810b65b00c62b76911b61c36cff3ec12ed9e801e6e2e976dcb WatchSource:0}: Error finding container b6a286b7a3c84e810b65b00c62b76911b61c36cff3ec12ed9e801e6e2e976dcb: Status 404 returned error can't find the container with id b6a286b7a3c84e810b65b00c62b76911b61c36cff3ec12ed9e801e6e2e976dcb Jul 16 12:19:41.097864 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:41.097848 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:19:41.586616 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:41.586578 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-mbxwh" event={"ID":"08da884b-cd39-43ed-bab1-62c7dfce73e8","Type":"ContainerStarted","Data":"b6a286b7a3c84e810b65b00c62b76911b61c36cff3ec12ed9e801e6e2e976dcb"} Jul 16 12:19:43.598065 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:43.598009 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-mbxwh" event={"ID":"08da884b-cd39-43ed-bab1-62c7dfce73e8","Type":"ContainerStarted","Data":"f06c7aeaf62bc8534a75929ab0fbc833ab54425892c227074f7fb3eeabdd8989"} Jul 16 12:19:43.598457 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:43.598183 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-mbxwh" Jul 16 12:19:43.618289 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:43.618237 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-mbxwh" podStartSLOduration=1.8575656980000002 podStartE2EDuration="3.618219788s" totalImagesPullingTime="1.76065409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:19:43.616359178 +0000 UTC m=+761.766066609" watchObservedRunningTime="2026-07-16 12:19:43.618219788 +0000 UTC m=+761.767927220" Jul 16 12:19:54.604856 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:19:54.604820 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-mbxwh" Jul 16 12:20:11.485767 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.485735 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr"] Jul 16 12:20:11.489268 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.489243 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.493778 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.493756 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 12:20:11.494086 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.494069 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hgfq8\"" Jul 16 12:20:11.495106 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.495085 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 12:20:11.495190 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.495128 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 16 12:20:11.506148 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.506121 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr"] Jul 16 12:20:11.580749 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.580709 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.580959 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.580792 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.580959 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.580827 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctfxl\" (UniqueName: \"kubernetes.io/projected/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-kube-api-access-ctfxl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.580959 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.580857 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.580959 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.580923 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.581128 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.580961 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.682100 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.682048 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.682100 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.682101 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.682348 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.682121 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.682348 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.682156 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.682348 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.682209 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.682348 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.682260 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctfxl\" (UniqueName: \"kubernetes.io/projected/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-kube-api-access-ctfxl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.682557 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.682476 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.682557 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.682535 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.682557 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.682553 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.684534 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.684508 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.684796 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.684779 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.691640 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.691613 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctfxl\" (UniqueName: \"kubernetes.io/projected/95e9b458-6e82-408d-a3a6-ee1c1fa42b19-kube-api-access-ctfxl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr\" (UID: \"95e9b458-6e82-408d-a3a6-ee1c1fa42b19\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.799371 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.799331 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:11.940126 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:11.940101 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr"] Jul 16 12:20:11.942962 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:20:11.942926 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e9b458_6e82_408d_a3a6_ee1c1fa42b19.slice/crio-68b88b2dc9df2f4a24e0eae7059fb517e03efd6c6d4b083d451ad93ada159947 WatchSource:0}: Error finding container 68b88b2dc9df2f4a24e0eae7059fb517e03efd6c6d4b083d451ad93ada159947: Status 404 returned error can't find the container with id 68b88b2dc9df2f4a24e0eae7059fb517e03efd6c6d4b083d451ad93ada159947 Jul 16 12:20:12.733117 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:12.733074 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" event={"ID":"95e9b458-6e82-408d-a3a6-ee1c1fa42b19","Type":"ContainerStarted","Data":"68b88b2dc9df2f4a24e0eae7059fb517e03efd6c6d4b083d451ad93ada159947"} Jul 16 12:20:13.742076 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:13.742039 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" event={"ID":"95e9b458-6e82-408d-a3a6-ee1c1fa42b19","Type":"ContainerStarted","Data":"2f21c137c1d051b076295ceb49974302a3999e93dc520e20614cb315f8b8734b"} Jul 16 12:20:13.763515 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:13.763459 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" podStartSLOduration=1.464499202 podStartE2EDuration="2.763442521s" totalImagesPullingTime="1.298943319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:20:13.761302545 +0000 UTC m=+791.911009978" watchObservedRunningTime="2026-07-16 12:20:13.763442521 +0000 UTC m=+791.913149952" Jul 16 12:20:21.799742 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:21.799698 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:21.800115 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:21.799772 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:21.812327 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:21.812298 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:22.792833 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:22.792801 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr" Jul 16 12:20:37.700222 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.700185 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td"] Jul 16 12:20:37.703717 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.703694 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.706514 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.706490 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 16 12:20:37.716504 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.716482 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td"] Jul 16 12:20:37.826351 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.826318 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmz76\" (UniqueName: \"kubernetes.io/projected/22b5d613-abe9-4d27-b954-e886cec96f01-kube-api-access-hmz76\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.826351 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.826356 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22b5d613-abe9-4d27-b954-e886cec96f01-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.826585 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.826427 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22b5d613-abe9-4d27-b954-e886cec96f01-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.826585 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.826459 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22b5d613-abe9-4d27-b954-e886cec96f01-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.826585 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.826499 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22b5d613-abe9-4d27-b954-e886cec96f01-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.826585 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.826525 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22b5d613-abe9-4d27-b954-e886cec96f01-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.927860 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.927819 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmz76\" (UniqueName: \"kubernetes.io/projected/22b5d613-abe9-4d27-b954-e886cec96f01-kube-api-access-hmz76\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.927860 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.927862 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22b5d613-abe9-4d27-b954-e886cec96f01-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.928107 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.927905 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22b5d613-abe9-4d27-b954-e886cec96f01-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.928107 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.927927 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22b5d613-abe9-4d27-b954-e886cec96f01-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.928107 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.927947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22b5d613-abe9-4d27-b954-e886cec96f01-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.928107 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.928096 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22b5d613-abe9-4d27-b954-e886cec96f01-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.928309 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.928293 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22b5d613-abe9-4d27-b954-e886cec96f01-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.928386 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.928362 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22b5d613-abe9-4d27-b954-e886cec96f01-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.928445 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.928398 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22b5d613-abe9-4d27-b954-e886cec96f01-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.930356 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.930333 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22b5d613-abe9-4d27-b954-e886cec96f01-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.930592 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.930574 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22b5d613-abe9-4d27-b954-e886cec96f01-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:37.945560 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:37.945532 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmz76\" (UniqueName: \"kubernetes.io/projected/22b5d613-abe9-4d27-b954-e886cec96f01-kube-api-access-hmz76\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td\" (UID: \"22b5d613-abe9-4d27-b954-e886cec96f01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:38.013649 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:38.013619 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:38.152120 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:38.152095 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td"] Jul 16 12:20:38.154029 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:20:38.153991 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b5d613_abe9_4d27_b954_e886cec96f01.slice/crio-2ebd931e47389e58e0b15c1b119004317be44992bb2407270342c659b3184dff WatchSource:0}: Error finding container 2ebd931e47389e58e0b15c1b119004317be44992bb2407270342c659b3184dff: Status 404 returned error can't find the container with id 2ebd931e47389e58e0b15c1b119004317be44992bb2407270342c659b3184dff Jul 16 12:20:38.854718 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:38.853654 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" event={"ID":"22b5d613-abe9-4d27-b954-e886cec96f01","Type":"ContainerStarted","Data":"a7b368223e4d5b51988681e04f5e5227730fbdf5e7641f3eeef99895d3d72b2e"} Jul 16 12:20:38.854718 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:38.853798 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" event={"ID":"22b5d613-abe9-4d27-b954-e886cec96f01","Type":"ContainerStarted","Data":"2ebd931e47389e58e0b15c1b119004317be44992bb2407270342c659b3184dff"} Jul 16 12:20:38.876706 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:38.876605 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" podStartSLOduration=1.676147094 podStartE2EDuration="1.876579457s" totalImagesPullingTime="200.432363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:20:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:20:38.874945568 +0000 UTC m=+817.024652995" watchObservedRunningTime="2026-07-16 12:20:38.876579457 +0000 UTC m=+817.026286879" Jul 16 12:20:48.014206 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:48.014158 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:48.014835 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:48.014265 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:48.026445 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:48.026418 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:48.914011 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:48.913964 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td" Jul 16 12:20:56.528646 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.528609 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds"] Jul 16 12:20:56.535640 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.535605 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.538849 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.538820 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 16 12:20:56.546495 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.546469 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds"] Jul 16 12:20:56.597939 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.597897 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f56f4e7d-b360-4313-8455-23d20f3c3b70-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.598129 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.598007 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f56f4e7d-b360-4313-8455-23d20f3c3b70-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.598129 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.598074 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f56f4e7d-b360-4313-8455-23d20f3c3b70-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.598210 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.598143 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f56f4e7d-b360-4313-8455-23d20f3c3b70-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.598210 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.598169 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2rm\" (UniqueName: \"kubernetes.io/projected/f56f4e7d-b360-4313-8455-23d20f3c3b70-kube-api-access-lf2rm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.598272 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.598243 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f56f4e7d-b360-4313-8455-23d20f3c3b70-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.698905 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.698858 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f56f4e7d-b360-4313-8455-23d20f3c3b70-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.699109 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.698913 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f56f4e7d-b360-4313-8455-23d20f3c3b70-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.699109 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.698943 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f56f4e7d-b360-4313-8455-23d20f3c3b70-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.699109 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.699059 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f56f4e7d-b360-4313-8455-23d20f3c3b70-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.699293 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.699129 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f56f4e7d-b360-4313-8455-23d20f3c3b70-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.699293 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.699164 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2rm\" (UniqueName: \"kubernetes.io/projected/f56f4e7d-b360-4313-8455-23d20f3c3b70-kube-api-access-lf2rm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.699429 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.699408 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f56f4e7d-b360-4313-8455-23d20f3c3b70-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.699483 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.699457 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f56f4e7d-b360-4313-8455-23d20f3c3b70-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.699535 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.699496 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f56f4e7d-b360-4313-8455-23d20f3c3b70-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.701128 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.701104 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f56f4e7d-b360-4313-8455-23d20f3c3b70-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.701386 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.701369 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f56f4e7d-b360-4313-8455-23d20f3c3b70-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.708671 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.708635 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2rm\" (UniqueName: \"kubernetes.io/projected/f56f4e7d-b360-4313-8455-23d20f3c3b70-kube-api-access-lf2rm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds\" (UID: \"f56f4e7d-b360-4313-8455-23d20f3c3b70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:56.847205 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:56.847122 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:20:57.012900 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:57.012870 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds"] Jul 16 12:20:57.015639 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:20:57.015605 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf56f4e7d_b360_4313_8455_23d20f3c3b70.slice/crio-25ff934c16bfb6dc611b1908ad201370b5d9d45475028856288b751b2f5c82cb WatchSource:0}: Error finding container 25ff934c16bfb6dc611b1908ad201370b5d9d45475028856288b751b2f5c82cb: Status 404 returned error can't find the container with id 25ff934c16bfb6dc611b1908ad201370b5d9d45475028856288b751b2f5c82cb Jul 16 12:20:57.950889 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:57.950850 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" event={"ID":"f56f4e7d-b360-4313-8455-23d20f3c3b70","Type":"ContainerStarted","Data":"8026bc0aaadadea501a6c6cfa0f91baf1a716ef195af4bed2ed12963b46487d5"} Jul 16 12:20:57.951279 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:57.950899 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" event={"ID":"f56f4e7d-b360-4313-8455-23d20f3c3b70","Type":"ContainerStarted","Data":"25ff934c16bfb6dc611b1908ad201370b5d9d45475028856288b751b2f5c82cb"} Jul 16 12:20:57.972560 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:20:57.972510 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" podStartSLOduration=1.780233347 podStartE2EDuration="1.972494368s" totalImagesPullingTime="192.261021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:20:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:20:57.970756826 +0000 UTC m=+836.120464256" watchObservedRunningTime="2026-07-16 12:20:57.972494368 +0000 UTC m=+836.122201799" Jul 16 12:21:06.848260 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:21:06.848218 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:21:06.848707 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:21:06.848307 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:21:06.860822 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:21:06.860790 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:21:07.006860 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:21:07.006828 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds" Jul 16 12:22:02.365318 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:02.365230 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:22:02.366438 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:02.366416 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:22:03.260034 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:03.259998 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7d676c7-h2t97"] Jul 16 12:22:03.274415 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:03.274366 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7d676c7-h2t97"] Jul 16 12:22:03.274594 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:03.274482 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7d676c7-h2t97" Jul 16 12:22:03.422462 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:03.422421 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/003e4c33-cbd4-4fab-8fe8-d4c9f418ddb0-tls-cert\") pod \"authorino-5dc7d676c7-h2t97\" (UID: \"003e4c33-cbd4-4fab-8fe8-d4c9f418ddb0\") " pod="kuadrant-system/authorino-5dc7d676c7-h2t97" Jul 16 12:22:03.422910 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:03.422497 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m4nd\" (UniqueName: \"kubernetes.io/projected/003e4c33-cbd4-4fab-8fe8-d4c9f418ddb0-kube-api-access-4m4nd\") pod \"authorino-5dc7d676c7-h2t97\" (UID: \"003e4c33-cbd4-4fab-8fe8-d4c9f418ddb0\") " pod="kuadrant-system/authorino-5dc7d676c7-h2t97" Jul 16 12:22:03.523607 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:03.523506 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m4nd\" (UniqueName: \"kubernetes.io/projected/003e4c33-cbd4-4fab-8fe8-d4c9f418ddb0-kube-api-access-4m4nd\") pod \"authorino-5dc7d676c7-h2t97\" (UID: \"003e4c33-cbd4-4fab-8fe8-d4c9f418ddb0\") " pod="kuadrant-system/authorino-5dc7d676c7-h2t97" Jul 16 12:22:03.523817 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:03.523631 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/003e4c33-cbd4-4fab-8fe8-d4c9f418ddb0-tls-cert\") pod \"authorino-5dc7d676c7-h2t97\" (UID: \"003e4c33-cbd4-4fab-8fe8-d4c9f418ddb0\") " pod="kuadrant-system/authorino-5dc7d676c7-h2t97" Jul 16 12:22:03.526099 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:03.526072 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/003e4c33-cbd4-4fab-8fe8-d4c9f418ddb0-tls-cert\") pod \"authorino-5dc7d676c7-h2t97\" (UID: \"003e4c33-cbd4-4fab-8fe8-d4c9f418ddb0\") " pod="kuadrant-system/authorino-5dc7d676c7-h2t97" Jul 16 12:22:03.532073 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:03.532053 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m4nd\" (UniqueName: \"kubernetes.io/projected/003e4c33-cbd4-4fab-8fe8-d4c9f418ddb0-kube-api-access-4m4nd\") pod \"authorino-5dc7d676c7-h2t97\" (UID: \"003e4c33-cbd4-4fab-8fe8-d4c9f418ddb0\") " pod="kuadrant-system/authorino-5dc7d676c7-h2t97" Jul 16 12:22:03.583542 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:03.583508 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7d676c7-h2t97" Jul 16 12:22:03.736312 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:03.736287 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7d676c7-h2t97"] Jul 16 12:22:03.738927 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:22:03.738899 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod003e4c33_cbd4_4fab_8fe8_d4c9f418ddb0.slice/crio-6a6074a9f2059a0f0b47fcba7f2b3a014628875fb75c9c3ea9a0c3d44b52083b WatchSource:0}: Error finding container 6a6074a9f2059a0f0b47fcba7f2b3a014628875fb75c9c3ea9a0c3d44b52083b: Status 404 returned error can't find the container with id 6a6074a9f2059a0f0b47fcba7f2b3a014628875fb75c9c3ea9a0c3d44b52083b Jul 16 12:22:04.280435 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:04.280387 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7d676c7-h2t97" event={"ID":"003e4c33-cbd4-4fab-8fe8-d4c9f418ddb0","Type":"ContainerStarted","Data":"98c612195388af61e4d533b2f7d8916094817f3cd4efcbcb8c0c847fa2db665f"} Jul 16 12:22:04.280643 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:04.280461 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7d676c7-h2t97" event={"ID":"003e4c33-cbd4-4fab-8fe8-d4c9f418ddb0","Type":"ContainerStarted","Data":"6a6074a9f2059a0f0b47fcba7f2b3a014628875fb75c9c3ea9a0c3d44b52083b"} Jul 16 12:22:04.299559 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:04.299493 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5dc7d676c7-h2t97" podStartSLOduration=0.986959192 podStartE2EDuration="1.299469988s" totalImagesPullingTime="312.510796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:22:04.298608168 +0000 UTC m=+902.448315617" watchObservedRunningTime="2026-07-16 12:22:04.299469988 +0000 UTC m=+902.449177421" Jul 16 12:22:04.337798 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:04.337578 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c765588b4-d6jxz"] Jul 16 12:22:04.337955 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:04.337846 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5c765588b4-d6jxz" podUID="fcea7d11-59e9-454c-99e0-5dee5da3823e" containerName="authorino" containerID="cri-o://9a085a2359c445d2bc74d46fa5712daec7c72f67981007fb370867aafc5cc9b4" gracePeriod=30 Jul 16 12:22:04.598102 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:04.598038 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c765588b4-d6jxz" Jul 16 12:22:04.736397 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:04.736355 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s24jq\" (UniqueName: \"kubernetes.io/projected/fcea7d11-59e9-454c-99e0-5dee5da3823e-kube-api-access-s24jq\") pod \"fcea7d11-59e9-454c-99e0-5dee5da3823e\" (UID: \"fcea7d11-59e9-454c-99e0-5dee5da3823e\") " Jul 16 12:22:04.736587 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:04.736458 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fcea7d11-59e9-454c-99e0-5dee5da3823e-tls-cert\") pod \"fcea7d11-59e9-454c-99e0-5dee5da3823e\" (UID: \"fcea7d11-59e9-454c-99e0-5dee5da3823e\") " Jul 16 12:22:04.738578 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:04.738540 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcea7d11-59e9-454c-99e0-5dee5da3823e-kube-api-access-s24jq" (OuterVolumeSpecName: "kube-api-access-s24jq") pod "fcea7d11-59e9-454c-99e0-5dee5da3823e" (UID: "fcea7d11-59e9-454c-99e0-5dee5da3823e"). InnerVolumeSpecName "kube-api-access-s24jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:22:04.746615 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:04.746585 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcea7d11-59e9-454c-99e0-5dee5da3823e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "fcea7d11-59e9-454c-99e0-5dee5da3823e" (UID: "fcea7d11-59e9-454c-99e0-5dee5da3823e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:22:04.837022 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:04.836992 2559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fcea7d11-59e9-454c-99e0-5dee5da3823e-tls-cert\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:22:04.837022 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:04.837020 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s24jq\" (UniqueName: \"kubernetes.io/projected/fcea7d11-59e9-454c-99e0-5dee5da3823e-kube-api-access-s24jq\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:22:05.287715 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:05.287668 2559 generic.go:356] "Generic (PLEG): container finished" podID="fcea7d11-59e9-454c-99e0-5dee5da3823e" containerID="9a085a2359c445d2bc74d46fa5712daec7c72f67981007fb370867aafc5cc9b4" exitCode=0 Jul 16 12:22:05.287908 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:05.287724 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c765588b4-d6jxz" Jul 16 12:22:05.287908 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:05.287761 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c765588b4-d6jxz" event={"ID":"fcea7d11-59e9-454c-99e0-5dee5da3823e","Type":"ContainerDied","Data":"9a085a2359c445d2bc74d46fa5712daec7c72f67981007fb370867aafc5cc9b4"} Jul 16 12:22:05.287908 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:05.287813 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c765588b4-d6jxz" event={"ID":"fcea7d11-59e9-454c-99e0-5dee5da3823e","Type":"ContainerDied","Data":"9d1f5025b04373978fdeb7384524d1fc2fa397e6beae1f3f5e695c33b81201cd"} Jul 16 12:22:05.287908 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:05.287845 2559 scope.go:117] "RemoveContainer" containerID="9a085a2359c445d2bc74d46fa5712daec7c72f67981007fb370867aafc5cc9b4" Jul 16 12:22:05.300654 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:05.300626 2559 scope.go:117] "RemoveContainer" containerID="9a085a2359c445d2bc74d46fa5712daec7c72f67981007fb370867aafc5cc9b4" Jul 16 12:22:05.301046 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:22:05.301015 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a085a2359c445d2bc74d46fa5712daec7c72f67981007fb370867aafc5cc9b4\": container with ID starting with 9a085a2359c445d2bc74d46fa5712daec7c72f67981007fb370867aafc5cc9b4 not found: ID does not exist" containerID="9a085a2359c445d2bc74d46fa5712daec7c72f67981007fb370867aafc5cc9b4" Jul 16 12:22:05.301144 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:05.301082 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a085a2359c445d2bc74d46fa5712daec7c72f67981007fb370867aafc5cc9b4"} err="failed to get container status \"9a085a2359c445d2bc74d46fa5712daec7c72f67981007fb370867aafc5cc9b4\": rpc error: code = NotFound desc = could not find container \"9a085a2359c445d2bc74d46fa5712daec7c72f67981007fb370867aafc5cc9b4\": container with ID starting with 9a085a2359c445d2bc74d46fa5712daec7c72f67981007fb370867aafc5cc9b4 not found: ID does not exist" Jul 16 12:22:05.320619 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:05.320584 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c765588b4-d6jxz"] Jul 16 12:22:05.329178 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:05.329145 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5c765588b4-d6jxz"] Jul 16 12:22:06.556746 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:22:06.556712 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcea7d11-59e9-454c-99e0-5dee5da3823e" path="/var/lib/kubelet/pods/fcea7d11-59e9-454c-99e0-5dee5da3823e/volumes" Jul 16 12:27:02.395499 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:27:02.395463 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:27:02.396975 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:27:02.396954 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:29:03.930356 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:03.930310 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s"] Jul 16 12:29:03.930965 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:03.930590 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" podUID="e86ef8d1-534c-4d8d-b07a-c5592c8311db" containerName="manager" containerID="cri-o://1c922ceabe279bb079b2170a7b3f4e2646deea38fcc3be8397bd5ac7678d7a03" gracePeriod=10 Jul 16 12:29:08.192862 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:08.192837 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" Jul 16 12:29:08.275026 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:08.274986 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpvj5\" (UniqueName: \"kubernetes.io/projected/e86ef8d1-534c-4d8d-b07a-c5592c8311db-kube-api-access-gpvj5\") pod \"e86ef8d1-534c-4d8d-b07a-c5592c8311db\" (UID: \"e86ef8d1-534c-4d8d-b07a-c5592c8311db\") " Jul 16 12:29:08.275216 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:08.275045 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e86ef8d1-534c-4d8d-b07a-c5592c8311db-extensions-socket-volume\") pod \"e86ef8d1-534c-4d8d-b07a-c5592c8311db\" (UID: \"e86ef8d1-534c-4d8d-b07a-c5592c8311db\") " Jul 16 12:29:08.275419 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:08.275394 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86ef8d1-534c-4d8d-b07a-c5592c8311db-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e86ef8d1-534c-4d8d-b07a-c5592c8311db" (UID: "e86ef8d1-534c-4d8d-b07a-c5592c8311db"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:29:08.277109 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:08.277087 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86ef8d1-534c-4d8d-b07a-c5592c8311db-kube-api-access-gpvj5" (OuterVolumeSpecName: "kube-api-access-gpvj5") pod "e86ef8d1-534c-4d8d-b07a-c5592c8311db" (UID: "e86ef8d1-534c-4d8d-b07a-c5592c8311db"). InnerVolumeSpecName "kube-api-access-gpvj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:29:08.334624 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:08.334583 2559 generic.go:356] "Generic (PLEG): container finished" podID="e86ef8d1-534c-4d8d-b07a-c5592c8311db" containerID="1c922ceabe279bb079b2170a7b3f4e2646deea38fcc3be8397bd5ac7678d7a03" exitCode=0 Jul 16 12:29:08.334830 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:08.334622 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" event={"ID":"e86ef8d1-534c-4d8d-b07a-c5592c8311db","Type":"ContainerDied","Data":"1c922ceabe279bb079b2170a7b3f4e2646deea38fcc3be8397bd5ac7678d7a03"} Jul 16 12:29:08.334830 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:08.334634 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" Jul 16 12:29:08.334830 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:08.334718 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s" event={"ID":"e86ef8d1-534c-4d8d-b07a-c5592c8311db","Type":"ContainerDied","Data":"b172a05146c43966be888c9c36e974537f4391035099fa0ad388c6ccd967bf8d"} Jul 16 12:29:08.334830 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:08.334752 2559 scope.go:117] "RemoveContainer" containerID="1c922ceabe279bb079b2170a7b3f4e2646deea38fcc3be8397bd5ac7678d7a03" Jul 16 12:29:08.344671 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:08.344647 2559 scope.go:117] "RemoveContainer" containerID="1c922ceabe279bb079b2170a7b3f4e2646deea38fcc3be8397bd5ac7678d7a03" Jul 16 12:29:08.344991 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:29:08.344966 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c922ceabe279bb079b2170a7b3f4e2646deea38fcc3be8397bd5ac7678d7a03\": container with ID starting with 1c922ceabe279bb079b2170a7b3f4e2646deea38fcc3be8397bd5ac7678d7a03 not found: ID does not exist" containerID="1c922ceabe279bb079b2170a7b3f4e2646deea38fcc3be8397bd5ac7678d7a03" Jul 16 12:29:08.345052 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:08.345001 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c922ceabe279bb079b2170a7b3f4e2646deea38fcc3be8397bd5ac7678d7a03"} err="failed to get container status \"1c922ceabe279bb079b2170a7b3f4e2646deea38fcc3be8397bd5ac7678d7a03\": rpc error: code = NotFound desc = could not find container \"1c922ceabe279bb079b2170a7b3f4e2646deea38fcc3be8397bd5ac7678d7a03\": container with ID starting with 1c922ceabe279bb079b2170a7b3f4e2646deea38fcc3be8397bd5ac7678d7a03 not found: ID does not exist" Jul 16 12:29:08.361933 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:08.361896 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s"] Jul 16 12:29:08.363561 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:08.363536 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jng5s"] Jul 16 12:29:08.376631 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:08.376600 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpvj5\" (UniqueName: \"kubernetes.io/projected/e86ef8d1-534c-4d8d-b07a-c5592c8311db-kube-api-access-gpvj5\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:29:08.376631 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:08.376628 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e86ef8d1-534c-4d8d-b07a-c5592c8311db-extensions-socket-volume\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:29:08.556703 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:29:08.556600 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e86ef8d1-534c-4d8d-b07a-c5592c8311db" path="/var/lib/kubelet/pods/e86ef8d1-534c-4d8d-b07a-c5592c8311db/volumes" Jul 16 12:30:00.228467 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:00.228430 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736750-cnsnd"] Jul 16 12:30:00.228993 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:00.228790 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e86ef8d1-534c-4d8d-b07a-c5592c8311db" containerName="manager" Jul 16 12:30:00.228993 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:00.228801 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e86ef8d1-534c-4d8d-b07a-c5592c8311db" containerName="manager" Jul 16 12:30:00.228993 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:00.228820 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcea7d11-59e9-454c-99e0-5dee5da3823e" containerName="authorino" Jul 16 12:30:00.228993 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:00.228825 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcea7d11-59e9-454c-99e0-5dee5da3823e" containerName="authorino" Jul 16 12:30:00.228993 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:00.228887 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e86ef8d1-534c-4d8d-b07a-c5592c8311db" containerName="manager" Jul 16 12:30:00.228993 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:00.228898 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcea7d11-59e9-454c-99e0-5dee5da3823e" containerName="authorino" Jul 16 12:30:00.232088 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:00.232070 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736750-cnsnd" Jul 16 12:30:00.236254 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:00.236228 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vxbx5\"" Jul 16 12:30:00.241489 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:00.241471 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736750-cnsnd" Jul 16 12:30:00.256350 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:00.256315 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736750-cnsnd"] Jul 16 12:30:00.399818 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:00.399778 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736750-cnsnd"] Jul 16 12:30:00.405057 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:30:00.405024 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56dc5ea0_b5dc_49d9_bd97_34ad199b3acc.slice/crio-c423c3d891d217f4f93694c430db8fd477a7b5fad6a9a4fd5f49d49ea8b29ca5 WatchSource:0}: Error finding container c423c3d891d217f4f93694c430db8fd477a7b5fad6a9a4fd5f49d49ea8b29ca5: Status 404 returned error can't find the container with id c423c3d891d217f4f93694c430db8fd477a7b5fad6a9a4fd5f49d49ea8b29ca5 Jul 16 12:30:00.407194 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:00.407178 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:30:00.575498 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:00.575459 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736750-cnsnd" event={"ID":"56dc5ea0-b5dc-49d9-bd97-34ad199b3acc","Type":"ContainerStarted","Data":"c423c3d891d217f4f93694c430db8fd477a7b5fad6a9a4fd5f49d49ea8b29ca5"} Jul 16 12:30:03.592143 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:03.592110 2559 generic.go:356] "Generic (PLEG): container finished" podID="56dc5ea0-b5dc-49d9-bd97-34ad199b3acc" containerID="35242280abe3a9375230092d61c28050e4476a969986d3ea6f966f12ff226e1d" exitCode=0 Jul 16 12:30:03.592483 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:03.592152 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736750-cnsnd" event={"ID":"56dc5ea0-b5dc-49d9-bd97-34ad199b3acc","Type":"ContainerDied","Data":"35242280abe3a9375230092d61c28050e4476a969986d3ea6f966f12ff226e1d"} Jul 16 12:30:04.726609 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:04.726576 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736750-cnsnd" Jul 16 12:30:05.289525 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.289486 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spmzx"] Jul 16 12:30:05.289880 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.289867 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56dc5ea0-b5dc-49d9-bd97-34ad199b3acc" containerName="cleanup" Jul 16 12:30:05.289929 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.289882 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="56dc5ea0-b5dc-49d9-bd97-34ad199b3acc" containerName="cleanup" Jul 16 12:30:05.289964 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.289959 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="56dc5ea0-b5dc-49d9-bd97-34ad199b3acc" containerName="cleanup" Jul 16 12:30:05.292969 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.292952 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spmzx" Jul 16 12:30:05.295841 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.295822 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-w7v5k\"" Jul 16 12:30:05.305824 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.305795 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spmzx"] Jul 16 12:30:05.383629 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.383583 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4gpc\" (UniqueName: \"kubernetes.io/projected/c222bb8c-3d5f-4ed6-84de-936443eaba9b-kube-api-access-k4gpc\") pod \"kuadrant-operator-controller-manager-86878c769f-spmzx\" (UID: \"c222bb8c-3d5f-4ed6-84de-936443eaba9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spmzx" Jul 16 12:30:05.383820 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.383754 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c222bb8c-3d5f-4ed6-84de-936443eaba9b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-spmzx\" (UID: \"c222bb8c-3d5f-4ed6-84de-936443eaba9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spmzx" Jul 16 12:30:05.484331 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.484285 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c222bb8c-3d5f-4ed6-84de-936443eaba9b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-spmzx\" (UID: \"c222bb8c-3d5f-4ed6-84de-936443eaba9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spmzx" Jul 16 12:30:05.484511 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.484446 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4gpc\" (UniqueName: \"kubernetes.io/projected/c222bb8c-3d5f-4ed6-84de-936443eaba9b-kube-api-access-k4gpc\") pod \"kuadrant-operator-controller-manager-86878c769f-spmzx\" (UID: \"c222bb8c-3d5f-4ed6-84de-936443eaba9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spmzx" Jul 16 12:30:05.484741 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.484718 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c222bb8c-3d5f-4ed6-84de-936443eaba9b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-spmzx\" (UID: \"c222bb8c-3d5f-4ed6-84de-936443eaba9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spmzx" Jul 16 12:30:05.493611 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.493588 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4gpc\" (UniqueName: \"kubernetes.io/projected/c222bb8c-3d5f-4ed6-84de-936443eaba9b-kube-api-access-k4gpc\") pod \"kuadrant-operator-controller-manager-86878c769f-spmzx\" (UID: \"c222bb8c-3d5f-4ed6-84de-936443eaba9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spmzx" Jul 16 12:30:05.602620 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.602535 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spmzx" Jul 16 12:30:05.602793 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.602774 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736750-cnsnd" Jul 16 12:30:05.602835 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.602820 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736750-cnsnd" event={"ID":"56dc5ea0-b5dc-49d9-bd97-34ad199b3acc","Type":"ContainerDied","Data":"c423c3d891d217f4f93694c430db8fd477a7b5fad6a9a4fd5f49d49ea8b29ca5"} Jul 16 12:30:05.602870 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.602860 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c423c3d891d217f4f93694c430db8fd477a7b5fad6a9a4fd5f49d49ea8b29ca5" Jul 16 12:30:05.758234 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:05.758201 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spmzx"] Jul 16 12:30:05.761132 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:30:05.761102 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc222bb8c_3d5f_4ed6_84de_936443eaba9b.slice/crio-b9322475ec72a3f6eb9b9b9db9695a6b0680d96ddb22ad06d82af4e258f34e32 WatchSource:0}: Error finding container b9322475ec72a3f6eb9b9b9db9695a6b0680d96ddb22ad06d82af4e258f34e32: Status 404 returned error can't find the container with id b9322475ec72a3f6eb9b9b9db9695a6b0680d96ddb22ad06d82af4e258f34e32 Jul 16 12:30:06.609181 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:06.609140 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spmzx" event={"ID":"c222bb8c-3d5f-4ed6-84de-936443eaba9b","Type":"ContainerStarted","Data":"67bb3c8a23ee4d55e74c0150b3fa8ab0c0de71d3599b99d839e592d6b430bd5b"} Jul 16 12:30:06.609392 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:06.609193 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spmzx" event={"ID":"c222bb8c-3d5f-4ed6-84de-936443eaba9b","Type":"ContainerStarted","Data":"b9322475ec72a3f6eb9b9b9db9695a6b0680d96ddb22ad06d82af4e258f34e32"} Jul 16 12:30:06.609392 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:06.609261 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spmzx" Jul 16 12:30:06.636118 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:06.636069 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spmzx" podStartSLOduration=1.63605278 podStartE2EDuration="1.63605278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:30:06.634545535 +0000 UTC m=+1384.784252995" watchObservedRunningTime="2026-07-16 12:30:06.63605278 +0000 UTC m=+1384.785760279" Jul 16 12:30:17.617297 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:30:17.617262 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spmzx" Jul 16 12:32:02.424398 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:32:02.424367 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:32:02.427699 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:32:02.427656 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:36:33.902051 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:33.901967 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4"] Jul 16 12:36:33.906068 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:33.906046 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4" Jul 16 12:36:33.909734 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:33.909705 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-051a7b7b-serving-cert\"" Jul 16 12:36:33.909927 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:33.909769 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x8mkx\"" Jul 16 12:36:33.909927 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:33.909791 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 12:36:33.910864 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:33.910839 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 12:36:33.919892 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:33.919861 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4"] Jul 16 12:36:33.976946 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:33.976915 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8980583-2084-44f9-a54d-ef294253fd9a-maas-api-tls\") pod \"maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4\" (UID: \"e8980583-2084-44f9-a54d-ef294253fd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4" Jul 16 12:36:33.977325 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:33.976976 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8980583-2084-44f9-a54d-ef294253fd9a-openshift-service-ca\") pod \"maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4\" (UID: \"e8980583-2084-44f9-a54d-ef294253fd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4" Jul 16 12:36:33.977325 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:33.977032 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmdwq\" (UniqueName: \"kubernetes.io/projected/e8980583-2084-44f9-a54d-ef294253fd9a-kube-api-access-gmdwq\") pod \"maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4\" (UID: \"e8980583-2084-44f9-a54d-ef294253fd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4" Jul 16 12:36:34.078040 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:34.078001 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8980583-2084-44f9-a54d-ef294253fd9a-maas-api-tls\") pod \"maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4\" (UID: \"e8980583-2084-44f9-a54d-ef294253fd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4" Jul 16 12:36:34.078230 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:34.078073 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8980583-2084-44f9-a54d-ef294253fd9a-openshift-service-ca\") pod \"maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4\" (UID: \"e8980583-2084-44f9-a54d-ef294253fd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4" Jul 16 12:36:34.078230 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:34.078117 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmdwq\" (UniqueName: \"kubernetes.io/projected/e8980583-2084-44f9-a54d-ef294253fd9a-kube-api-access-gmdwq\") pod \"maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4\" (UID: \"e8980583-2084-44f9-a54d-ef294253fd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4" Jul 16 12:36:34.078798 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:34.078772 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8980583-2084-44f9-a54d-ef294253fd9a-openshift-service-ca\") pod \"maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4\" (UID: \"e8980583-2084-44f9-a54d-ef294253fd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4" Jul 16 12:36:34.080710 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:34.080668 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8980583-2084-44f9-a54d-ef294253fd9a-maas-api-tls\") pod \"maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4\" (UID: \"e8980583-2084-44f9-a54d-ef294253fd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4" Jul 16 12:36:34.089254 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:34.089230 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmdwq\" (UniqueName: \"kubernetes.io/projected/e8980583-2084-44f9-a54d-ef294253fd9a-kube-api-access-gmdwq\") pod \"maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4\" (UID: \"e8980583-2084-44f9-a54d-ef294253fd9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4" Jul 16 12:36:34.217737 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:34.217623 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4" Jul 16 12:36:34.360589 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:34.360564 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4"] Jul 16 12:36:34.362690 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:36:34.362641 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8980583_2084_44f9_a54d_ef294253fd9a.slice/crio-76a80ce8a722c8eedff0a48911c052a0c2127172e568aeb4140f17a825cec881 WatchSource:0}: Error finding container 76a80ce8a722c8eedff0a48911c052a0c2127172e568aeb4140f17a825cec881: Status 404 returned error can't find the container with id 76a80ce8a722c8eedff0a48911c052a0c2127172e568aeb4140f17a825cec881 Jul 16 12:36:34.364219 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:34.364194 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:36:34.531594 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:34.531557 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4" event={"ID":"e8980583-2084-44f9-a54d-ef294253fd9a","Type":"ContainerStarted","Data":"76a80ce8a722c8eedff0a48911c052a0c2127172e568aeb4140f17a825cec881"} Jul 16 12:36:37.548885 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:37.548855 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4" event={"ID":"e8980583-2084-44f9-a54d-ef294253fd9a","Type":"ContainerStarted","Data":"d72f545757ea4ef6f0288b42112f4de1e5ed4fddc79e24ebf3b87220a9d9a98a"} Jul 16 12:36:37.549276 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:37.548925 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4" Jul 16 12:36:37.569366 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:37.569314 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4" podStartSLOduration=2.4069846630000002 podStartE2EDuration="4.569298491s" totalImagesPullingTime="2.162313828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:36:37.568114833 +0000 UTC m=+1775.717822268" watchObservedRunningTime="2026-07-16 12:36:37.569298491 +0000 UTC m=+1775.719005924" Jul 16 12:36:43.560153 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:36:43.560120 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4" Jul 16 12:37:02.454161 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:37:02.454120 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:37:02.457610 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:37:02.457588 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:38:43.514027 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:43.513987 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk"] Jul 16 12:38:43.518118 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:43.518096 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk" Jul 16 12:38:43.521800 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:43.521771 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-df8d3cf1-serving-cert\"" Jul 16 12:38:43.534743 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:43.534712 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk"] Jul 16 12:38:43.573865 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:43.573825 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm9wt\" (UniqueName: \"kubernetes.io/projected/edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1-kube-api-access-rm9wt\") pod \"maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk\" (UID: \"edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk" Jul 16 12:38:43.574070 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:43.573935 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1-openshift-service-ca\") pod \"maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk\" (UID: \"edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk" Jul 16 12:38:43.574070 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:43.573958 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1-maas-api-tls\") pod \"maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk\" (UID: \"edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk" Jul 16 12:38:43.674952 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:43.674916 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1-openshift-service-ca\") pod \"maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk\" (UID: \"edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk" Jul 16 12:38:43.674952 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:43.674958 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1-maas-api-tls\") pod \"maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk\" (UID: \"edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk" Jul 16 12:38:43.675194 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:43.674995 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm9wt\" (UniqueName: \"kubernetes.io/projected/edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1-kube-api-access-rm9wt\") pod \"maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk\" (UID: \"edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk" Jul 16 12:38:43.675588 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:43.675568 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1-openshift-service-ca\") pod \"maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk\" (UID: \"edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk" Jul 16 12:38:43.677489 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:43.677467 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1-maas-api-tls\") pod \"maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk\" (UID: \"edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk" Jul 16 12:38:43.683943 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:43.683906 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm9wt\" (UniqueName: \"kubernetes.io/projected/edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1-kube-api-access-rm9wt\") pod \"maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk\" (UID: \"edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk" Jul 16 12:38:43.828536 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:43.828443 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk" Jul 16 12:38:43.975533 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:43.975502 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk"] Jul 16 12:38:43.977143 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:38:43.977113 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf93aa4_bb5c_4d56_9e7d_cfb1149f13d1.slice/crio-1ccea5742482b1b3a9f934bfa1ed14d3cffd4330a96cc5bc1f26af912788e7b5 WatchSource:0}: Error finding container 1ccea5742482b1b3a9f934bfa1ed14d3cffd4330a96cc5bc1f26af912788e7b5: Status 404 returned error can't find the container with id 1ccea5742482b1b3a9f934bfa1ed14d3cffd4330a96cc5bc1f26af912788e7b5 Jul 16 12:38:44.160050 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:44.159953 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk" event={"ID":"edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1","Type":"ContainerStarted","Data":"1ccea5742482b1b3a9f934bfa1ed14d3cffd4330a96cc5bc1f26af912788e7b5"} Jul 16 12:38:45.167008 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:45.166964 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk" event={"ID":"edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1","Type":"ContainerStarted","Data":"bbd2f74e856775fb9cce7b06f0dafa64f80feff51da852a6703601188d78d576"} Jul 16 12:38:45.167417 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:45.167055 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk" Jul 16 12:38:45.188277 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:45.188211 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk" podStartSLOduration=1.759136547 podStartE2EDuration="2.188194693s" totalImagesPullingTime="429.058146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:38:45.186318665 +0000 UTC m=+1903.336026128" watchObservedRunningTime="2026-07-16 12:38:45.188194693 +0000 UTC m=+1903.337902126" Jul 16 12:38:51.178798 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:38:51.178770 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk" Jul 16 12:40:26.411109 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:26.411066 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn"] Jul 16 12:40:26.415329 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:26.415310 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn" Jul 16 12:40:26.418627 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:26.418589 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-28d017-serving-cert\"" Jul 16 12:40:26.426626 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:26.426597 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn"] Jul 16 12:40:26.532237 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:26.532192 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62b5a0fe-3d54-409c-8581-2c85bcf83923-openshift-service-ca\") pod \"maas-api-e2e-inf-a-28d017-64948b66f7-59svn\" (UID: \"62b5a0fe-3d54-409c-8581-2c85bcf83923\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn" Jul 16 12:40:26.532418 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:26.532256 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62b5a0fe-3d54-409c-8581-2c85bcf83923-maas-api-tls\") pod \"maas-api-e2e-inf-a-28d017-64948b66f7-59svn\" (UID: \"62b5a0fe-3d54-409c-8581-2c85bcf83923\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn" Jul 16 12:40:26.532418 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:26.532275 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6lpg\" (UniqueName: \"kubernetes.io/projected/62b5a0fe-3d54-409c-8581-2c85bcf83923-kube-api-access-t6lpg\") pod \"maas-api-e2e-inf-a-28d017-64948b66f7-59svn\" (UID: \"62b5a0fe-3d54-409c-8581-2c85bcf83923\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn" Jul 16 12:40:26.633405 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:26.633358 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62b5a0fe-3d54-409c-8581-2c85bcf83923-openshift-service-ca\") pod \"maas-api-e2e-inf-a-28d017-64948b66f7-59svn\" (UID: \"62b5a0fe-3d54-409c-8581-2c85bcf83923\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn" Jul 16 12:40:26.633637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:26.633428 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62b5a0fe-3d54-409c-8581-2c85bcf83923-maas-api-tls\") pod \"maas-api-e2e-inf-a-28d017-64948b66f7-59svn\" (UID: \"62b5a0fe-3d54-409c-8581-2c85bcf83923\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn" Jul 16 12:40:26.633637 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:26.633451 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6lpg\" (UniqueName: \"kubernetes.io/projected/62b5a0fe-3d54-409c-8581-2c85bcf83923-kube-api-access-t6lpg\") pod \"maas-api-e2e-inf-a-28d017-64948b66f7-59svn\" (UID: \"62b5a0fe-3d54-409c-8581-2c85bcf83923\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn" Jul 16 12:40:26.633637 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:40:26.633535 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-serving-cert: secret "maas-api-e2e-inf-a-28d017-serving-cert" not found Jul 16 12:40:26.633637 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:40:26.633616 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b5a0fe-3d54-409c-8581-2c85bcf83923-maas-api-tls podName:62b5a0fe-3d54-409c-8581-2c85bcf83923 nodeName:}" failed. No retries permitted until 2026-07-16 12:40:27.133596018 +0000 UTC m=+2005.283303448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/62b5a0fe-3d54-409c-8581-2c85bcf83923-maas-api-tls") pod "maas-api-e2e-inf-a-28d017-64948b66f7-59svn" (UID: "62b5a0fe-3d54-409c-8581-2c85bcf83923") : secret "maas-api-e2e-inf-a-28d017-serving-cert" not found Jul 16 12:40:26.634250 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:26.634225 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62b5a0fe-3d54-409c-8581-2c85bcf83923-openshift-service-ca\") pod \"maas-api-e2e-inf-a-28d017-64948b66f7-59svn\" (UID: \"62b5a0fe-3d54-409c-8581-2c85bcf83923\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn" Jul 16 12:40:26.646640 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:26.646600 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6lpg\" (UniqueName: \"kubernetes.io/projected/62b5a0fe-3d54-409c-8581-2c85bcf83923-kube-api-access-t6lpg\") pod \"maas-api-e2e-inf-a-28d017-64948b66f7-59svn\" (UID: \"62b5a0fe-3d54-409c-8581-2c85bcf83923\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn" Jul 16 12:40:27.138516 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:27.138478 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62b5a0fe-3d54-409c-8581-2c85bcf83923-maas-api-tls\") pod \"maas-api-e2e-inf-a-28d017-64948b66f7-59svn\" (UID: \"62b5a0fe-3d54-409c-8581-2c85bcf83923\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn" Jul 16 12:40:27.140988 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:27.140954 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62b5a0fe-3d54-409c-8581-2c85bcf83923-maas-api-tls\") pod \"maas-api-e2e-inf-a-28d017-64948b66f7-59svn\" (UID: \"62b5a0fe-3d54-409c-8581-2c85bcf83923\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn" Jul 16 12:40:27.327112 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:27.327064 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn" Jul 16 12:40:27.477497 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:27.477469 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn"] Jul 16 12:40:27.480518 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:40:27.480490 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b5a0fe_3d54_409c_8581_2c85bcf83923.slice/crio-349084d67f1703d3b8a30647468199085c7ec04790af7abb74fbbc9df76018fb WatchSource:0}: Error finding container 349084d67f1703d3b8a30647468199085c7ec04790af7abb74fbbc9df76018fb: Status 404 returned error can't find the container with id 349084d67f1703d3b8a30647468199085c7ec04790af7abb74fbbc9df76018fb Jul 16 12:40:27.661120 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:27.661078 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn" event={"ID":"62b5a0fe-3d54-409c-8581-2c85bcf83923","Type":"ContainerStarted","Data":"349084d67f1703d3b8a30647468199085c7ec04790af7abb74fbbc9df76018fb"} Jul 16 12:40:28.668446 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:28.668408 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn" event={"ID":"62b5a0fe-3d54-409c-8581-2c85bcf83923","Type":"ContainerStarted","Data":"4e261e5c2f7721e29078438214b53ed5fdb84f1dabb9eee2a816597f6bf7b947"} Jul 16 12:40:28.668952 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:28.668613 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn" Jul 16 12:40:28.694989 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:28.694936 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn" podStartSLOduration=2.048930896 podStartE2EDuration="2.694917178s" totalImagesPullingTime="645.986282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:40:28.69249743 +0000 UTC m=+2006.842204863" watchObservedRunningTime="2026-07-16 12:40:28.694917178 +0000 UTC m=+2006.844624611" Jul 16 12:40:34.680089 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:34.680052 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d017-64948b66f7-59svn" Jul 16 12:40:35.087744 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:35.087709 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5"] Jul 16 12:40:35.091506 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:35.091484 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5" Jul 16 12:40:35.094810 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:35.094785 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9ac0a7-serving-cert\"" Jul 16 12:40:35.113050 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:35.113011 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5"] Jul 16 12:40:35.213281 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:35.213244 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a02f00-ddb3-4fa6-9193-4b27a2df8e19-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5\" (UID: \"f1a02f00-ddb3-4fa6-9193-4b27a2df8e19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5" Jul 16 12:40:35.213457 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:35.213294 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnrmw\" (UniqueName: \"kubernetes.io/projected/f1a02f00-ddb3-4fa6-9193-4b27a2df8e19-kube-api-access-bnrmw\") pod \"maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5\" (UID: \"f1a02f00-ddb3-4fa6-9193-4b27a2df8e19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5" Jul 16 12:40:35.213457 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:35.213330 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1a02f00-ddb3-4fa6-9193-4b27a2df8e19-maas-api-tls\") pod \"maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5\" (UID: \"f1a02f00-ddb3-4fa6-9193-4b27a2df8e19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5" Jul 16 12:40:35.314198 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:35.314152 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a02f00-ddb3-4fa6-9193-4b27a2df8e19-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5\" (UID: \"f1a02f00-ddb3-4fa6-9193-4b27a2df8e19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5" Jul 16 12:40:35.314380 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:35.314213 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnrmw\" (UniqueName: \"kubernetes.io/projected/f1a02f00-ddb3-4fa6-9193-4b27a2df8e19-kube-api-access-bnrmw\") pod \"maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5\" (UID: \"f1a02f00-ddb3-4fa6-9193-4b27a2df8e19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5" Jul 16 12:40:35.314380 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:35.314232 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1a02f00-ddb3-4fa6-9193-4b27a2df8e19-maas-api-tls\") pod \"maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5\" (UID: \"f1a02f00-ddb3-4fa6-9193-4b27a2df8e19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5" Jul 16 12:40:35.315038 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:35.315013 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a02f00-ddb3-4fa6-9193-4b27a2df8e19-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5\" (UID: \"f1a02f00-ddb3-4fa6-9193-4b27a2df8e19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5" Jul 16 12:40:35.316729 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:35.316698 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1a02f00-ddb3-4fa6-9193-4b27a2df8e19-maas-api-tls\") pod \"maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5\" (UID: \"f1a02f00-ddb3-4fa6-9193-4b27a2df8e19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5" Jul 16 12:40:35.325494 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:35.325463 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnrmw\" (UniqueName: \"kubernetes.io/projected/f1a02f00-ddb3-4fa6-9193-4b27a2df8e19-kube-api-access-bnrmw\") pod \"maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5\" (UID: \"f1a02f00-ddb3-4fa6-9193-4b27a2df8e19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5" Jul 16 12:40:35.403382 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:35.403287 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5" Jul 16 12:40:35.549766 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:35.549738 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5"] Jul 16 12:40:35.551050 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:40:35.551018 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a02f00_ddb3_4fa6_9193_4b27a2df8e19.slice/crio-6f7cf7334a2af47f81234e7c6926bd26f71288c444cbce33ee7e37f704ad1e9d WatchSource:0}: Error finding container 6f7cf7334a2af47f81234e7c6926bd26f71288c444cbce33ee7e37f704ad1e9d: Status 404 returned error can't find the container with id 6f7cf7334a2af47f81234e7c6926bd26f71288c444cbce33ee7e37f704ad1e9d Jul 16 12:40:35.703481 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:35.703390 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5" event={"ID":"f1a02f00-ddb3-4fa6-9193-4b27a2df8e19","Type":"ContainerStarted","Data":"6f7cf7334a2af47f81234e7c6926bd26f71288c444cbce33ee7e37f704ad1e9d"} Jul 16 12:40:36.711257 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:36.711218 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5" event={"ID":"f1a02f00-ddb3-4fa6-9193-4b27a2df8e19","Type":"ContainerStarted","Data":"7c07b9ce23d486c350be7cdfe4674d6d3f66866397c5d35527e2ebba587148ae"} Jul 16 12:40:36.711665 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:36.711356 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5" Jul 16 12:40:36.731842 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:36.731783 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5" podStartSLOduration=1.429186047 podStartE2EDuration="1.731765244s" totalImagesPullingTime="302.579197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:40:36.730664466 +0000 UTC m=+2014.880371901" watchObservedRunningTime="2026-07-16 12:40:36.731765244 +0000 UTC m=+2014.881472677" Jul 16 12:40:42.721456 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:42.721425 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5" Jul 16 12:40:56.115908 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.115864 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv"] Jul 16 12:40:56.120217 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.120199 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.124317 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.124290 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-28d017\"/\"openshift-service-ca.crt\"" Jul 16 12:40:56.124469 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.124290 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-28d017\"/\"default-dockercfg-5l8gk\"" Jul 16 12:40:56.124469 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.124291 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-28d017\"/\"kube-root-ca.crt\"" Jul 16 12:40:56.124469 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.124329 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-28d017\"/\"test-model-28d017-kserve-self-signed-certs\"" Jul 16 12:40:56.130517 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.130488 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv"] Jul 16 12:40:56.216071 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.216035 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd82335-205d-4501-8408-9a43cde197d4-tls-certs\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.216255 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.216078 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-model-cache\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.216255 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.216137 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-tmp-dir\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.216255 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.216159 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-home\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.216255 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.216217 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-dshm\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.216255 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.216242 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5dbv\" (UniqueName: \"kubernetes.io/projected/1fd82335-205d-4501-8408-9a43cde197d4-kube-api-access-t5dbv\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.317359 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.317323 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd82335-205d-4501-8408-9a43cde197d4-tls-certs\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.317359 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.317364 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-model-cache\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.317580 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.317396 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-tmp-dir\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.317580 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.317414 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-home\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.317580 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.317440 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-dshm\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.317580 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.317461 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5dbv\" (UniqueName: \"kubernetes.io/projected/1fd82335-205d-4501-8408-9a43cde197d4-kube-api-access-t5dbv\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.317949 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.317916 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-home\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.318032 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.317927 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-tmp-dir\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.318032 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.317963 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-model-cache\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.319842 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.319819 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-dshm\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.320052 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.320037 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd82335-205d-4501-8408-9a43cde197d4-tls-certs\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.327629 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.327606 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5dbv\" (UniqueName: \"kubernetes.io/projected/1fd82335-205d-4501-8408-9a43cde197d4-kube-api-access-t5dbv\") pod \"test-model-28d017-kserve-bd9c4dc69-frmdv\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.431575 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.431476 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:40:56.579874 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.579844 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv"] Jul 16 12:40:56.581922 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:40:56.581895 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fd82335_205d_4501_8408_9a43cde197d4.slice/crio-828094ba2828421e3efd1945fc83e5485705a253f6e61ca444c903670f743c09 WatchSource:0}: Error finding container 828094ba2828421e3efd1945fc83e5485705a253f6e61ca444c903670f743c09: Status 404 returned error can't find the container with id 828094ba2828421e3efd1945fc83e5485705a253f6e61ca444c903670f743c09 Jul 16 12:40:56.813329 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.813293 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" event={"ID":"1fd82335-205d-4501-8408-9a43cde197d4","Type":"ContainerStarted","Data":"1f1827681975e4f541cdd4fec76e927571f341c6603b7fbe529725112cabf4a3"} Jul 16 12:40:56.813503 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.813336 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" event={"ID":"1fd82335-205d-4501-8408-9a43cde197d4","Type":"ContainerStarted","Data":"828094ba2828421e3efd1945fc83e5485705a253f6e61ca444c903670f743c09"} Jul 16 12:40:56.834520 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:40:56.834455 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" podStartSLOduration=0.834437699 podStartE2EDuration="834.437699ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:40:56.832481449 +0000 UTC m=+2034.982188883" watchObservedRunningTime="2026-07-16 12:40:56.834437699 +0000 UTC m=+2034.984145131" Jul 16 12:41:06.431824 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:06.431792 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:41:06.432290 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:06.432012 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:41:06.444430 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:06.444403 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:41:06.877520 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:06.877481 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:41:23.492561 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.492522 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg"] Jul 16 12:41:23.495984 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.495966 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.498986 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.498962 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9ac0a7\"/\"kube-root-ca.crt\"" Jul 16 12:41:23.499106 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.498990 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9ac0a7\"/\"test-model-9ac0a7-kserve-self-signed-certs\"" Jul 16 12:41:23.499106 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.498967 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9ac0a7\"/\"default-dockercfg-wfszb\"" Jul 16 12:41:23.500066 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.500046 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9ac0a7\"/\"openshift-service-ca.crt\"" Jul 16 12:41:23.509246 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.509222 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg"] Jul 16 12:41:23.581735 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.581673 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f9db61e-8512-4a67-b3f1-968b339afce5-tls-certs\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.581919 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.581759 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-home\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.581919 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.581856 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-tmp-dir\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.582022 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.581931 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-dshm\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.582058 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.582027 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-model-cache\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.582100 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.582063 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxwp7\" (UniqueName: \"kubernetes.io/projected/6f9db61e-8512-4a67-b3f1-968b339afce5-kube-api-access-zxwp7\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.683504 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.683465 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-dshm\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.683750 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.683515 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-model-cache\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.683750 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.683537 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxwp7\" (UniqueName: \"kubernetes.io/projected/6f9db61e-8512-4a67-b3f1-968b339afce5-kube-api-access-zxwp7\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.683750 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.683559 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f9db61e-8512-4a67-b3f1-968b339afce5-tls-certs\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.683750 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.683595 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-home\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.683750 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.683655 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-tmp-dir\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.684027 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.684011 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-model-cache\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.684093 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.684072 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-tmp-dir\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.684140 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.684115 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-home\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.686037 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.686016 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-dshm\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.686789 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.686772 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f9db61e-8512-4a67-b3f1-968b339afce5-tls-certs\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.692534 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.692512 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxwp7\" (UniqueName: \"kubernetes.io/projected/6f9db61e-8512-4a67-b3f1-968b339afce5-kube-api-access-zxwp7\") pod \"test-model-9ac0a7-kserve-6b887fccc9-l6wdg\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.805741 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.805617 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:23.949213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:23.949189 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg"] Jul 16 12:41:23.951406 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:41:23.951382 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f9db61e_8512_4a67_b3f1_968b339afce5.slice/crio-9818ab47d030555b698ed73b654eff28983a573eacafeceb93996c99ef3508a1 WatchSource:0}: Error finding container 9818ab47d030555b698ed73b654eff28983a573eacafeceb93996c99ef3508a1: Status 404 returned error can't find the container with id 9818ab47d030555b698ed73b654eff28983a573eacafeceb93996c99ef3508a1 Jul 16 12:41:24.954470 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:24.954431 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" event={"ID":"6f9db61e-8512-4a67-b3f1-968b339afce5","Type":"ContainerStarted","Data":"d3e3e481fc23fea31d7b9d30846ca1728ce1676e098f2f8baf0f093cf02ea1fc"} Jul 16 12:41:24.954470 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:24.954478 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" event={"ID":"6f9db61e-8512-4a67-b3f1-968b339afce5","Type":"ContainerStarted","Data":"9818ab47d030555b698ed73b654eff28983a573eacafeceb93996c99ef3508a1"} Jul 16 12:41:24.974009 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:24.973756 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" podStartSLOduration=1.973735108 podStartE2EDuration="1.973735108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:41:24.97326213 +0000 UTC m=+2063.122969563" watchObservedRunningTime="2026-07-16 12:41:24.973735108 +0000 UTC m=+2063.123442543" Jul 16 12:41:33.806217 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:33.806153 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:33.806217 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:33.806227 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:33.819344 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:33.819316 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:34.011313 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:34.011283 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:41:48.275882 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:48.275845 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv"] Jul 16 12:41:49.350716 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:41:49.350666 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg"] Jul 16 12:42:02.485969 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:02.485937 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:42:02.489275 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:02.489251 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:42:03.296174 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.296117 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" podUID="1fd82335-205d-4501-8408-9a43cde197d4" containerName="main" containerID="cri-o://1f1827681975e4f541cdd4fec76e927571f341c6603b7fbe529725112cabf4a3" gracePeriod=45 Jul 16 12:42:03.550095 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.550030 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:42:03.667428 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.667394 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-tmp-dir\") pod \"1fd82335-205d-4501-8408-9a43cde197d4\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " Jul 16 12:42:03.667634 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.667449 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-model-cache\") pod \"1fd82335-205d-4501-8408-9a43cde197d4\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " Jul 16 12:42:03.667634 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.667490 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5dbv\" (UniqueName: \"kubernetes.io/projected/1fd82335-205d-4501-8408-9a43cde197d4-kube-api-access-t5dbv\") pod \"1fd82335-205d-4501-8408-9a43cde197d4\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " Jul 16 12:42:03.667634 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.667512 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd82335-205d-4501-8408-9a43cde197d4-tls-certs\") pod \"1fd82335-205d-4501-8408-9a43cde197d4\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " Jul 16 12:42:03.667634 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.667542 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-home\") pod \"1fd82335-205d-4501-8408-9a43cde197d4\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " Jul 16 12:42:03.667634 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.667588 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-dshm\") pod \"1fd82335-205d-4501-8408-9a43cde197d4\" (UID: \"1fd82335-205d-4501-8408-9a43cde197d4\") " Jul 16 12:42:03.667918 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.667734 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1fd82335-205d-4501-8408-9a43cde197d4" (UID: "1fd82335-205d-4501-8408-9a43cde197d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:42:03.667981 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.667743 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "1fd82335-205d-4501-8408-9a43cde197d4" (UID: "1fd82335-205d-4501-8408-9a43cde197d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:42:03.668041 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.668007 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-home" (OuterVolumeSpecName: "home") pod "1fd82335-205d-4501-8408-9a43cde197d4" (UID: "1fd82335-205d-4501-8408-9a43cde197d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:42:03.668107 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.668079 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-tmp-dir\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:42:03.668155 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.668114 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-model-cache\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:42:03.669921 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.669889 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-dshm" (OuterVolumeSpecName: "dshm") pod "1fd82335-205d-4501-8408-9a43cde197d4" (UID: "1fd82335-205d-4501-8408-9a43cde197d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:42:03.670088 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.670068 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd82335-205d-4501-8408-9a43cde197d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1fd82335-205d-4501-8408-9a43cde197d4" (UID: "1fd82335-205d-4501-8408-9a43cde197d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:42:03.670154 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.670097 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd82335-205d-4501-8408-9a43cde197d4-kube-api-access-t5dbv" (OuterVolumeSpecName: "kube-api-access-t5dbv") pod "1fd82335-205d-4501-8408-9a43cde197d4" (UID: "1fd82335-205d-4501-8408-9a43cde197d4"). InnerVolumeSpecName "kube-api-access-t5dbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:42:03.768549 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.768516 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5dbv\" (UniqueName: \"kubernetes.io/projected/1fd82335-205d-4501-8408-9a43cde197d4-kube-api-access-t5dbv\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:42:03.768549 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.768546 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd82335-205d-4501-8408-9a43cde197d4-tls-certs\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:42:03.768549 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.768557 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-home\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:42:03.768817 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:03.768564 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd82335-205d-4501-8408-9a43cde197d4-dshm\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:42:04.157819 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.157783 2559 generic.go:356] "Generic (PLEG): container finished" podID="1fd82335-205d-4501-8408-9a43cde197d4" containerID="1f1827681975e4f541cdd4fec76e927571f341c6603b7fbe529725112cabf4a3" exitCode=0 Jul 16 12:42:04.157819 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.157822 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" Jul 16 12:42:04.158101 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.157861 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" event={"ID":"1fd82335-205d-4501-8408-9a43cde197d4","Type":"ContainerDied","Data":"1f1827681975e4f541cdd4fec76e927571f341c6603b7fbe529725112cabf4a3"} Jul 16 12:42:04.158101 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.157911 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv" event={"ID":"1fd82335-205d-4501-8408-9a43cde197d4","Type":"ContainerDied","Data":"828094ba2828421e3efd1945fc83e5485705a253f6e61ca444c903670f743c09"} Jul 16 12:42:04.158101 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.157935 2559 scope.go:117] "RemoveContainer" containerID="1f1827681975e4f541cdd4fec76e927571f341c6603b7fbe529725112cabf4a3" Jul 16 12:42:04.168494 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.168476 2559 scope.go:117] "RemoveContainer" containerID="1f1827681975e4f541cdd4fec76e927571f341c6603b7fbe529725112cabf4a3" Jul 16 12:42:04.168869 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:42:04.168844 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f1827681975e4f541cdd4fec76e927571f341c6603b7fbe529725112cabf4a3\": container with ID starting with 1f1827681975e4f541cdd4fec76e927571f341c6603b7fbe529725112cabf4a3 not found: ID does not exist" containerID="1f1827681975e4f541cdd4fec76e927571f341c6603b7fbe529725112cabf4a3" Jul 16 12:42:04.168961 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.168876 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f1827681975e4f541cdd4fec76e927571f341c6603b7fbe529725112cabf4a3"} err="failed to get container status \"1f1827681975e4f541cdd4fec76e927571f341c6603b7fbe529725112cabf4a3\": rpc error: code = NotFound desc = could not find container \"1f1827681975e4f541cdd4fec76e927571f341c6603b7fbe529725112cabf4a3\": container with ID starting with 1f1827681975e4f541cdd4fec76e927571f341c6603b7fbe529725112cabf4a3 not found: ID does not exist" Jul 16 12:42:04.180399 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.180375 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv"] Jul 16 12:42:04.186015 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.185991 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-28d017/test-model-28d017-kserve-bd9c4dc69-frmdv"] Jul 16 12:42:04.371394 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.371333 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" podUID="6f9db61e-8512-4a67-b3f1-968b339afce5" containerName="main" containerID="cri-o://d3e3e481fc23fea31d7b9d30846ca1728ce1676e098f2f8baf0f093cf02ea1fc" gracePeriod=45 Jul 16 12:42:04.557827 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.557794 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fd82335-205d-4501-8408-9a43cde197d4" path="/var/lib/kubelet/pods/1fd82335-205d-4501-8408-9a43cde197d4/volumes" Jul 16 12:42:04.628530 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.628504 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:42:04.676226 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.676191 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxwp7\" (UniqueName: \"kubernetes.io/projected/6f9db61e-8512-4a67-b3f1-968b339afce5-kube-api-access-zxwp7\") pod \"6f9db61e-8512-4a67-b3f1-968b339afce5\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " Jul 16 12:42:04.676426 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.676253 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-dshm\") pod \"6f9db61e-8512-4a67-b3f1-968b339afce5\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " Jul 16 12:42:04.676426 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.676288 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-home\") pod \"6f9db61e-8512-4a67-b3f1-968b339afce5\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " Jul 16 12:42:04.676426 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.676318 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f9db61e-8512-4a67-b3f1-968b339afce5-tls-certs\") pod \"6f9db61e-8512-4a67-b3f1-968b339afce5\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " Jul 16 12:42:04.676426 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.676391 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-tmp-dir\") pod \"6f9db61e-8512-4a67-b3f1-968b339afce5\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " Jul 16 12:42:04.676426 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.676421 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-model-cache\") pod \"6f9db61e-8512-4a67-b3f1-968b339afce5\" (UID: \"6f9db61e-8512-4a67-b3f1-968b339afce5\") " Jul 16 12:42:04.676750 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.676719 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-home" (OuterVolumeSpecName: "home") pod "6f9db61e-8512-4a67-b3f1-968b339afce5" (UID: "6f9db61e-8512-4a67-b3f1-968b339afce5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:42:04.676882 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.676857 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f9db61e-8512-4a67-b3f1-968b339afce5" (UID: "6f9db61e-8512-4a67-b3f1-968b339afce5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:42:04.677534 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.677496 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f9db61e-8512-4a67-b3f1-968b339afce5" (UID: "6f9db61e-8512-4a67-b3f1-968b339afce5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:42:04.678927 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.678904 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9db61e-8512-4a67-b3f1-968b339afce5-kube-api-access-zxwp7" (OuterVolumeSpecName: "kube-api-access-zxwp7") pod "6f9db61e-8512-4a67-b3f1-968b339afce5" (UID: "6f9db61e-8512-4a67-b3f1-968b339afce5"). InnerVolumeSpecName "kube-api-access-zxwp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:42:04.678927 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.678905 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-dshm" (OuterVolumeSpecName: "dshm") pod "6f9db61e-8512-4a67-b3f1-968b339afce5" (UID: "6f9db61e-8512-4a67-b3f1-968b339afce5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:42:04.678927 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.678907 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f9db61e-8512-4a67-b3f1-968b339afce5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f9db61e-8512-4a67-b3f1-968b339afce5" (UID: "6f9db61e-8512-4a67-b3f1-968b339afce5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:42:04.777800 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.777768 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-dshm\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:42:04.777800 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.777795 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-home\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:42:04.777800 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.777803 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f9db61e-8512-4a67-b3f1-968b339afce5-tls-certs\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:42:04.777800 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.777813 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-tmp-dir\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:42:04.778069 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.777821 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f9db61e-8512-4a67-b3f1-968b339afce5-model-cache\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:42:04.778069 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:04.777830 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxwp7\" (UniqueName: \"kubernetes.io/projected/6f9db61e-8512-4a67-b3f1-968b339afce5-kube-api-access-zxwp7\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:42:05.164859 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:05.164763 2559 generic.go:356] "Generic (PLEG): container finished" podID="6f9db61e-8512-4a67-b3f1-968b339afce5" containerID="d3e3e481fc23fea31d7b9d30846ca1728ce1676e098f2f8baf0f093cf02ea1fc" exitCode=0 Jul 16 12:42:05.164859 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:05.164804 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" Jul 16 12:42:05.164859 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:05.164838 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" event={"ID":"6f9db61e-8512-4a67-b3f1-968b339afce5","Type":"ContainerDied","Data":"d3e3e481fc23fea31d7b9d30846ca1728ce1676e098f2f8baf0f093cf02ea1fc"} Jul 16 12:42:05.165127 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:05.164883 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg" event={"ID":"6f9db61e-8512-4a67-b3f1-968b339afce5","Type":"ContainerDied","Data":"9818ab47d030555b698ed73b654eff28983a573eacafeceb93996c99ef3508a1"} Jul 16 12:42:05.165127 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:05.164903 2559 scope.go:117] "RemoveContainer" containerID="d3e3e481fc23fea31d7b9d30846ca1728ce1676e098f2f8baf0f093cf02ea1fc" Jul 16 12:42:05.174545 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:05.174526 2559 scope.go:117] "RemoveContainer" containerID="d3e3e481fc23fea31d7b9d30846ca1728ce1676e098f2f8baf0f093cf02ea1fc" Jul 16 12:42:05.174837 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:42:05.174820 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e3e481fc23fea31d7b9d30846ca1728ce1676e098f2f8baf0f093cf02ea1fc\": container with ID starting with d3e3e481fc23fea31d7b9d30846ca1728ce1676e098f2f8baf0f093cf02ea1fc not found: ID does not exist" containerID="d3e3e481fc23fea31d7b9d30846ca1728ce1676e098f2f8baf0f093cf02ea1fc" Jul 16 12:42:05.174909 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:05.174845 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e3e481fc23fea31d7b9d30846ca1728ce1676e098f2f8baf0f093cf02ea1fc"} err="failed to get container status \"d3e3e481fc23fea31d7b9d30846ca1728ce1676e098f2f8baf0f093cf02ea1fc\": rpc error: code = NotFound desc = could not find container \"d3e3e481fc23fea31d7b9d30846ca1728ce1676e098f2f8baf0f093cf02ea1fc\": container with ID starting with d3e3e481fc23fea31d7b9d30846ca1728ce1676e098f2f8baf0f093cf02ea1fc not found: ID does not exist" Jul 16 12:42:05.187943 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:05.187915 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg"] Jul 16 12:42:05.191739 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:05.191713 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9ac0a7/test-model-9ac0a7-kserve-6b887fccc9-l6wdg"] Jul 16 12:42:06.557339 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:06.557296 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f9db61e-8512-4a67-b3f1-968b339afce5" path="/var/lib/kubelet/pods/6f9db61e-8512-4a67-b3f1-968b339afce5/volumes" Jul 16 12:42:52.216602 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.216566 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p"] Jul 16 12:42:52.217283 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.217265 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f9db61e-8512-4a67-b3f1-968b339afce5" containerName="main" Jul 16 12:42:52.217327 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.217286 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f9db61e-8512-4a67-b3f1-968b339afce5" containerName="main" Jul 16 12:42:52.217327 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.217301 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fd82335-205d-4501-8408-9a43cde197d4" containerName="main" Jul 16 12:42:52.217327 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.217309 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fd82335-205d-4501-8408-9a43cde197d4" containerName="main" Jul 16 12:42:52.217437 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.217426 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fd82335-205d-4501-8408-9a43cde197d4" containerName="main" Jul 16 12:42:52.217480 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.217443 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f9db61e-8512-4a67-b3f1-968b339afce5" containerName="main" Jul 16 12:42:52.220743 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.220721 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p" Jul 16 12:42:52.224088 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.224056 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-47e820-serving-cert\"" Jul 16 12:42:52.230025 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.230001 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p"] Jul 16 12:42:52.238331 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.238304 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sndn4\" (UniqueName: \"kubernetes.io/projected/544647d1-97b6-4f1f-8856-7afc0d35d1b9-kube-api-access-sndn4\") pod \"maas-api-e2e-api-b-47e820-778f585dff-44r9p\" (UID: \"544647d1-97b6-4f1f-8856-7afc0d35d1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p" Jul 16 12:42:52.238463 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.238338 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/544647d1-97b6-4f1f-8856-7afc0d35d1b9-maas-api-tls\") pod \"maas-api-e2e-api-b-47e820-778f585dff-44r9p\" (UID: \"544647d1-97b6-4f1f-8856-7afc0d35d1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p" Jul 16 12:42:52.238505 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.238468 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/544647d1-97b6-4f1f-8856-7afc0d35d1b9-openshift-service-ca\") pod \"maas-api-e2e-api-b-47e820-778f585dff-44r9p\" (UID: \"544647d1-97b6-4f1f-8856-7afc0d35d1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p" Jul 16 12:42:52.339082 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.339049 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sndn4\" (UniqueName: \"kubernetes.io/projected/544647d1-97b6-4f1f-8856-7afc0d35d1b9-kube-api-access-sndn4\") pod \"maas-api-e2e-api-b-47e820-778f585dff-44r9p\" (UID: \"544647d1-97b6-4f1f-8856-7afc0d35d1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p" Jul 16 12:42:52.339082 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.339087 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/544647d1-97b6-4f1f-8856-7afc0d35d1b9-maas-api-tls\") pod \"maas-api-e2e-api-b-47e820-778f585dff-44r9p\" (UID: \"544647d1-97b6-4f1f-8856-7afc0d35d1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p" Jul 16 12:42:52.339321 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.339155 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/544647d1-97b6-4f1f-8856-7afc0d35d1b9-openshift-service-ca\") pod \"maas-api-e2e-api-b-47e820-778f585dff-44r9p\" (UID: \"544647d1-97b6-4f1f-8856-7afc0d35d1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p" Jul 16 12:42:52.340086 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.340061 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/544647d1-97b6-4f1f-8856-7afc0d35d1b9-openshift-service-ca\") pod \"maas-api-e2e-api-b-47e820-778f585dff-44r9p\" (UID: \"544647d1-97b6-4f1f-8856-7afc0d35d1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p" Jul 16 12:42:52.341910 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.341879 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/544647d1-97b6-4f1f-8856-7afc0d35d1b9-maas-api-tls\") pod \"maas-api-e2e-api-b-47e820-778f585dff-44r9p\" (UID: \"544647d1-97b6-4f1f-8856-7afc0d35d1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p" Jul 16 12:42:52.348062 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.348039 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sndn4\" (UniqueName: \"kubernetes.io/projected/544647d1-97b6-4f1f-8856-7afc0d35d1b9-kube-api-access-sndn4\") pod \"maas-api-e2e-api-b-47e820-778f585dff-44r9p\" (UID: \"544647d1-97b6-4f1f-8856-7afc0d35d1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p" Jul 16 12:42:52.533562 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.533515 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p" Jul 16 12:42:52.684775 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.684729 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p"] Jul 16 12:42:52.691275 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:52.691240 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:42:53.403440 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:53.403403 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p" event={"ID":"544647d1-97b6-4f1f-8856-7afc0d35d1b9","Type":"ContainerStarted","Data":"4b1a7ef8102791e440c27322497ca065fdfa90c24f26ef54bd3583b6cbdef6bc"} Jul 16 12:42:54.410919 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:54.410884 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p" event={"ID":"544647d1-97b6-4f1f-8856-7afc0d35d1b9","Type":"ContainerStarted","Data":"c76ccfd5aa6a5d3d37ee38297e17b116f8d292ea7baaae76169e6e2fd7506f89"} Jul 16 12:42:54.411375 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:54.410957 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p" Jul 16 12:42:54.434038 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:42:54.433963 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p" podStartSLOduration=1.481272084 podStartE2EDuration="2.433946094s" totalImagesPullingTime="952.67401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:42:54.431092446 +0000 UTC m=+2152.580799878" watchObservedRunningTime="2026-07-16 12:42:54.433946094 +0000 UTC m=+2152.583653526" Jul 16 12:43:00.420731 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:43:00.420672 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47e820-778f585dff-44r9p" Jul 16 12:44:36.782206 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:36.782171 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz"] Jul 16 12:44:36.785966 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:36.785944 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:36.790360 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:36.790336 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3c72a3\"/\"auth-test-model-3c72a3-kserve-self-signed-certs\"" Jul 16 12:44:36.790713 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:36.790363 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3c72a3\"/\"kube-root-ca.crt\"" Jul 16 12:44:36.790821 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:36.790447 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3c72a3\"/\"default-dockercfg-j2g8z\"" Jul 16 12:44:36.790894 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:36.790821 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3c72a3\"/\"openshift-service-ca.crt\"" Jul 16 12:44:36.798491 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:36.798466 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz"] Jul 16 12:44:36.961575 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:36.961532 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-tmp-dir\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:36.961818 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:36.961608 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-dshm\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:36.961818 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:36.961634 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-model-cache\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:36.961818 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:36.961750 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djcfx\" (UniqueName: \"kubernetes.io/projected/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-kube-api-access-djcfx\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:36.961818 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:36.961782 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-home\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:36.962005 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:36.961822 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-tls-certs\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:37.062653 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:37.062571 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-tls-certs\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:37.062653 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:37.062639 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-tmp-dir\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:37.062898 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:37.062708 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-dshm\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:37.062898 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:37.062728 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-model-cache\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:37.062898 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:37.062797 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djcfx\" (UniqueName: \"kubernetes.io/projected/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-kube-api-access-djcfx\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:37.062898 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:37.062834 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-home\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:37.063215 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:37.063185 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-tmp-dir\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:37.063394 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:37.063225 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-home\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:37.063525 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:37.063267 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-model-cache\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:37.065566 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:37.065538 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-dshm\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:37.065964 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:37.065940 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-tls-certs\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:37.072346 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:37.072321 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djcfx\" (UniqueName: \"kubernetes.io/projected/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-kube-api-access-djcfx\") pod \"auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:37.096948 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:37.096916 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:37.241971 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:37.241946 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz"] Jul 16 12:44:37.243846 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:44:37.243818 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddca32236_a342_4e37_a2ed_e0c4beb5a0a0.slice/crio-7f3f3bd2e9dbe0fe7e20b8a8b1992c801bc4ace66ec77283a81ec80fb161633b WatchSource:0}: Error finding container 7f3f3bd2e9dbe0fe7e20b8a8b1992c801bc4ace66ec77283a81ec80fb161633b: Status 404 returned error can't find the container with id 7f3f3bd2e9dbe0fe7e20b8a8b1992c801bc4ace66ec77283a81ec80fb161633b Jul 16 12:44:37.923126 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:37.923089 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" event={"ID":"dca32236-a342-4e37-a2ed-e0c4beb5a0a0","Type":"ContainerStarted","Data":"6e86791e695f6c7116d11e41473924f56aa3e07e29f10c79ec45f9e11d90400f"} Jul 16 12:44:37.923508 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:37.923136 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" event={"ID":"dca32236-a342-4e37-a2ed-e0c4beb5a0a0","Type":"ContainerStarted","Data":"7f3f3bd2e9dbe0fe7e20b8a8b1992c801bc4ace66ec77283a81ec80fb161633b"} Jul 16 12:44:37.947306 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:37.947252 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" podStartSLOduration=1.9472352179999999 podStartE2EDuration="1.947235218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:44:37.944250599 +0000 UTC m=+2256.093958031" watchObservedRunningTime="2026-07-16 12:44:37.947235218 +0000 UTC m=+2256.096942651" Jul 16 12:44:47.097534 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:47.097492 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:47.098133 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:47.097559 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:47.109875 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:47.109849 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:44:47.988516 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:44:47.988486 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:45:00.203101 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:00.203064 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736765-fz7ts"] Jul 16 12:45:00.206893 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:00.206871 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736765-fz7ts" Jul 16 12:45:00.214537 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:00.214146 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vxbx5\"" Jul 16 12:45:00.217500 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:00.217484 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736765-fz7ts" Jul 16 12:45:00.238695 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:00.238642 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736765-fz7ts"] Jul 16 12:45:00.246342 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:00.246311 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9ac0a7-29736765-bxt5n"] Jul 16 12:45:00.250999 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:00.250973 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9ac0a7-29736765-bxt5n" Jul 16 12:45:00.252780 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:00.252751 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9ac0a7-29736765-bxt5n"] Jul 16 12:45:00.265341 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:00.265309 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9ac0a7-29736765-bxt5n" Jul 16 12:45:00.387625 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:00.387579 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736765-fz7ts"] Jul 16 12:45:00.391177 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:45:00.391145 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a13dea_5090_48ee_9dba_4fc4c28a3d76.slice/crio-87ea2a6de382de502c2f1dfb0d22f250ab492ac032c61bb8a7d0314f3d04325f WatchSource:0}: Error finding container 87ea2a6de382de502c2f1dfb0d22f250ab492ac032c61bb8a7d0314f3d04325f: Status 404 returned error can't find the container with id 87ea2a6de382de502c2f1dfb0d22f250ab492ac032c61bb8a7d0314f3d04325f Jul 16 12:45:00.417100 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:00.417074 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9ac0a7-29736765-bxt5n"] Jul 16 12:45:00.422859 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:45:00.422831 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88bfe16a_308c_459b_a870_7da4b6437df7.slice/crio-0f9abededeb2ed420fabff2b24b6c466572947441dc2a5ba3d39523523157c14 WatchSource:0}: Error finding container 0f9abededeb2ed420fabff2b24b6c466572947441dc2a5ba3d39523523157c14: Status 404 returned error can't find the container with id 0f9abededeb2ed420fabff2b24b6c466572947441dc2a5ba3d39523523157c14 Jul 16 12:45:01.047647 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:01.047554 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736765-fz7ts" event={"ID":"49a13dea-5090-48ee-9dba-4fc4c28a3d76","Type":"ContainerStarted","Data":"321251d2b68ac20ac9c928588db00b0e3b0ea62931097e98cc8a48b72d93b0d2"} Jul 16 12:45:01.047647 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:01.047609 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736765-fz7ts" event={"ID":"49a13dea-5090-48ee-9dba-4fc4c28a3d76","Type":"ContainerStarted","Data":"87ea2a6de382de502c2f1dfb0d22f250ab492ac032c61bb8a7d0314f3d04325f"} Jul 16 12:45:01.049112 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:01.049080 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9ac0a7-29736765-bxt5n" event={"ID":"88bfe16a-308c-459b-a870-7da4b6437df7","Type":"ContainerStarted","Data":"45f46adee4d7e8452187fb0f2da7ff843af531cab00fef8684e211aa01c6da7a"} Jul 16 12:45:01.049242 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:01.049130 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9ac0a7-29736765-bxt5n" event={"ID":"88bfe16a-308c-459b-a870-7da4b6437df7","Type":"ContainerStarted","Data":"0f9abededeb2ed420fabff2b24b6c466572947441dc2a5ba3d39523523157c14"} Jul 16 12:45:01.063250 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:01.063058 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736765-fz7ts" podStartSLOduration=1.063039562 podStartE2EDuration="1.063039562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:01.062451578 +0000 UTC m=+2279.212159011" watchObservedRunningTime="2026-07-16 12:45:01.063039562 +0000 UTC m=+2279.212746996" Jul 16 12:45:01.079663 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:01.079611 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9ac0a7-29736765-bxt5n" podStartSLOduration=1.079593326 podStartE2EDuration="1.079593326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:01.078258995 +0000 UTC m=+2279.227966428" watchObservedRunningTime="2026-07-16 12:45:01.079593326 +0000 UTC m=+2279.229300782" Jul 16 12:45:02.055257 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:02.055220 2559 generic.go:356] "Generic (PLEG): container finished" podID="49a13dea-5090-48ee-9dba-4fc4c28a3d76" containerID="321251d2b68ac20ac9c928588db00b0e3b0ea62931097e98cc8a48b72d93b0d2" exitCode=0 Jul 16 12:45:02.055743 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:02.055294 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736765-fz7ts" event={"ID":"49a13dea-5090-48ee-9dba-4fc4c28a3d76","Type":"ContainerDied","Data":"321251d2b68ac20ac9c928588db00b0e3b0ea62931097e98cc8a48b72d93b0d2"} Jul 16 12:45:02.056720 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:02.056662 2559 generic.go:356] "Generic (PLEG): container finished" podID="88bfe16a-308c-459b-a870-7da4b6437df7" containerID="45f46adee4d7e8452187fb0f2da7ff843af531cab00fef8684e211aa01c6da7a" exitCode=0 Jul 16 12:45:02.056839 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:02.056745 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9ac0a7-29736765-bxt5n" event={"ID":"88bfe16a-308c-459b-a870-7da4b6437df7","Type":"ContainerDied","Data":"45f46adee4d7e8452187fb0f2da7ff843af531cab00fef8684e211aa01c6da7a"} Jul 16 12:45:03.203638 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:03.203609 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9ac0a7-29736765-bxt5n" Jul 16 12:45:03.247860 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:03.247831 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736765-fz7ts" Jul 16 12:45:04.069159 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:04.069130 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736765-fz7ts" Jul 16 12:45:04.069352 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:04.069127 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736765-fz7ts" event={"ID":"49a13dea-5090-48ee-9dba-4fc4c28a3d76","Type":"ContainerDied","Data":"87ea2a6de382de502c2f1dfb0d22f250ab492ac032c61bb8a7d0314f3d04325f"} Jul 16 12:45:04.069352 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:04.069259 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87ea2a6de382de502c2f1dfb0d22f250ab492ac032c61bb8a7d0314f3d04325f" Jul 16 12:45:04.074259 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:04.074215 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9ac0a7-29736765-bxt5n" event={"ID":"88bfe16a-308c-459b-a870-7da4b6437df7","Type":"ContainerDied","Data":"0f9abededeb2ed420fabff2b24b6c466572947441dc2a5ba3d39523523157c14"} Jul 16 12:45:04.074436 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:04.074315 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f9abededeb2ed420fabff2b24b6c466572947441dc2a5ba3d39523523157c14" Jul 16 12:45:04.074436 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:45:04.074394 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9ac0a7-29736765-bxt5n" Jul 16 12:46:51.600344 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:46:51.600308 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz"] Jul 16 12:47:02.516912 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:02.516880 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:47:02.520915 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:02.520890 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mwt4_b64ae5fe-696a-4fc9-8a0b-249efff10f28/ovn-acl-logging/0.log" Jul 16 12:47:06.620764 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:06.620699 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" podUID="dca32236-a342-4e37-a2ed-e0c4beb5a0a0" containerName="main" containerID="cri-o://6e86791e695f6c7116d11e41473924f56aa3e07e29f10c79ec45f9e11d90400f" gracePeriod=45 Jul 16 12:47:06.879969 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:06.879914 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:47:07.015776 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.015739 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-home\") pod \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " Jul 16 12:47:07.015776 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.015784 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-model-cache\") pod \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " Jul 16 12:47:07.016031 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.015822 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djcfx\" (UniqueName: \"kubernetes.io/projected/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-kube-api-access-djcfx\") pod \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " Jul 16 12:47:07.016031 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.015938 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-dshm\") pod \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " Jul 16 12:47:07.016031 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.015966 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-tls-certs\") pod \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " Jul 16 12:47:07.016031 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.016024 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-tmp-dir\") pod \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\" (UID: \"dca32236-a342-4e37-a2ed-e0c4beb5a0a0\") " Jul 16 12:47:07.016231 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.016078 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "dca32236-a342-4e37-a2ed-e0c4beb5a0a0" (UID: "dca32236-a342-4e37-a2ed-e0c4beb5a0a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:47:07.016231 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.016089 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-home" (OuterVolumeSpecName: "home") pod "dca32236-a342-4e37-a2ed-e0c4beb5a0a0" (UID: "dca32236-a342-4e37-a2ed-e0c4beb5a0a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:47:07.016358 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.016334 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dca32236-a342-4e37-a2ed-e0c4beb5a0a0" (UID: "dca32236-a342-4e37-a2ed-e0c4beb5a0a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:47:07.016413 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.016344 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-home\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:47:07.016413 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.016382 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-model-cache\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:47:07.018155 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.018117 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-dshm" (OuterVolumeSpecName: "dshm") pod "dca32236-a342-4e37-a2ed-e0c4beb5a0a0" (UID: "dca32236-a342-4e37-a2ed-e0c4beb5a0a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:47:07.018262 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.018199 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-kube-api-access-djcfx" (OuterVolumeSpecName: "kube-api-access-djcfx") pod "dca32236-a342-4e37-a2ed-e0c4beb5a0a0" (UID: "dca32236-a342-4e37-a2ed-e0c4beb5a0a0"). InnerVolumeSpecName "kube-api-access-djcfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:47:07.018262 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.018201 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dca32236-a342-4e37-a2ed-e0c4beb5a0a0" (UID: "dca32236-a342-4e37-a2ed-e0c4beb5a0a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:47:07.117264 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.117218 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-dshm\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:47:07.117264 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.117256 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-tls-certs\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:47:07.117264 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.117269 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-tmp-dir\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:47:07.117504 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.117281 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djcfx\" (UniqueName: \"kubernetes.io/projected/dca32236-a342-4e37-a2ed-e0c4beb5a0a0-kube-api-access-djcfx\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:47:07.708215 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.708182 2559 generic.go:356] "Generic (PLEG): container finished" podID="dca32236-a342-4e37-a2ed-e0c4beb5a0a0" containerID="6e86791e695f6c7116d11e41473924f56aa3e07e29f10c79ec45f9e11d90400f" exitCode=0 Jul 16 12:47:07.708777 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.708226 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" Jul 16 12:47:07.708777 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.708280 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" event={"ID":"dca32236-a342-4e37-a2ed-e0c4beb5a0a0","Type":"ContainerDied","Data":"6e86791e695f6c7116d11e41473924f56aa3e07e29f10c79ec45f9e11d90400f"} Jul 16 12:47:07.708777 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.708332 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz" event={"ID":"dca32236-a342-4e37-a2ed-e0c4beb5a0a0","Type":"ContainerDied","Data":"7f3f3bd2e9dbe0fe7e20b8a8b1992c801bc4ace66ec77283a81ec80fb161633b"} Jul 16 12:47:07.708777 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.708357 2559 scope.go:117] "RemoveContainer" containerID="6e86791e695f6c7116d11e41473924f56aa3e07e29f10c79ec45f9e11d90400f" Jul 16 12:47:07.718464 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.718441 2559 scope.go:117] "RemoveContainer" containerID="6e86791e695f6c7116d11e41473924f56aa3e07e29f10c79ec45f9e11d90400f" Jul 16 12:47:07.718749 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:47:07.718731 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e86791e695f6c7116d11e41473924f56aa3e07e29f10c79ec45f9e11d90400f\": container with ID starting with 6e86791e695f6c7116d11e41473924f56aa3e07e29f10c79ec45f9e11d90400f not found: ID does not exist" containerID="6e86791e695f6c7116d11e41473924f56aa3e07e29f10c79ec45f9e11d90400f" Jul 16 12:47:07.718809 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.718757 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e86791e695f6c7116d11e41473924f56aa3e07e29f10c79ec45f9e11d90400f"} err="failed to get container status \"6e86791e695f6c7116d11e41473924f56aa3e07e29f10c79ec45f9e11d90400f\": rpc error: code = NotFound desc = could not find container \"6e86791e695f6c7116d11e41473924f56aa3e07e29f10c79ec45f9e11d90400f\": container with ID starting with 6e86791e695f6c7116d11e41473924f56aa3e07e29f10c79ec45f9e11d90400f not found: ID does not exist" Jul 16 12:47:07.733244 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.733218 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz"] Jul 16 12:47:07.738922 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:07.738896 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/auth-test-model-3c72a3-kserve-58b77d6cf-8bxdz"] Jul 16 12:47:08.557832 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:08.557793 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dca32236-a342-4e37-a2ed-e0c4beb5a0a0" path="/var/lib/kubelet/pods/dca32236-a342-4e37-a2ed-e0c4beb5a0a0/volumes" Jul 16 12:47:27.376193 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.376157 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj"] Jul 16 12:47:27.376738 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.376653 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dca32236-a342-4e37-a2ed-e0c4beb5a0a0" containerName="main" Jul 16 12:47:27.376738 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.376665 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dca32236-a342-4e37-a2ed-e0c4beb5a0a0" containerName="main" Jul 16 12:47:27.376738 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.376708 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49a13dea-5090-48ee-9dba-4fc4c28a3d76" containerName="cleanup" Jul 16 12:47:27.376738 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.376715 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="49a13dea-5090-48ee-9dba-4fc4c28a3d76" containerName="cleanup" Jul 16 12:47:27.376738 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.376724 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88bfe16a-308c-459b-a870-7da4b6437df7" containerName="cleanup" Jul 16 12:47:27.376738 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.376731 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="88bfe16a-308c-459b-a870-7da4b6437df7" containerName="cleanup" Jul 16 12:47:27.377049 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.376791 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="88bfe16a-308c-459b-a870-7da4b6437df7" containerName="cleanup" Jul 16 12:47:27.377049 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.376801 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dca32236-a342-4e37-a2ed-e0c4beb5a0a0" containerName="main" Jul 16 12:47:27.377049 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.376808 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="49a13dea-5090-48ee-9dba-4fc4c28a3d76" containerName="cleanup" Jul 16 12:47:27.379933 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.379913 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.383870 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.383844 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3c72a3\"/\"default-dockercfg-j2g8z\"" Jul 16 12:47:27.383999 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.383922 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3c72a3\"/\"openshift-service-ca.crt\"" Jul 16 12:47:27.383999 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.383974 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3c72a3\"/\"sub-test-model-3c72a3-kserve-self-signed-certs\"" Jul 16 12:47:27.384121 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.384063 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3c72a3\"/\"kube-root-ca.crt\"" Jul 16 12:47:27.392280 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.392244 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj"] Jul 16 12:47:27.411166 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.411126 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf5fs\" (UniqueName: \"kubernetes.io/projected/1c3129e8-fd1f-4d19-9112-1048e92a0645-kube-api-access-rf5fs\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.411371 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.411195 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-home\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.411371 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.411229 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-dshm\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.411371 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.411258 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3129e8-fd1f-4d19-9112-1048e92a0645-tls-certs\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.411371 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.411285 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-model-cache\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.411580 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.411424 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-tmp-dir\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.512015 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.511971 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-tmp-dir\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.512219 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.512050 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf5fs\" (UniqueName: \"kubernetes.io/projected/1c3129e8-fd1f-4d19-9112-1048e92a0645-kube-api-access-rf5fs\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.512219 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.512098 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-home\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.512219 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.512126 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-dshm\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.512219 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.512154 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3129e8-fd1f-4d19-9112-1048e92a0645-tls-certs\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.512219 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.512179 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-model-cache\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.512498 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.512457 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-tmp-dir\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.512560 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.512506 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-home\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.512646 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.512626 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-model-cache\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.514391 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.514363 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-dshm\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.514722 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.514703 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3129e8-fd1f-4d19-9112-1048e92a0645-tls-certs\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.521453 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.521432 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf5fs\" (UniqueName: \"kubernetes.io/projected/1c3129e8-fd1f-4d19-9112-1048e92a0645-kube-api-access-rf5fs\") pod \"sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.690600 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.690495 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:27.830629 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:27.830601 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj"] Jul 16 12:47:27.832229 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:47:27.832199 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c3129e8_fd1f_4d19_9112_1048e92a0645.slice/crio-25dd0843032db4a002da5a142a8f0a78d971830393ba106222e0cd3cbb354501 WatchSource:0}: Error finding container 25dd0843032db4a002da5a142a8f0a78d971830393ba106222e0cd3cbb354501: Status 404 returned error can't find the container with id 25dd0843032db4a002da5a142a8f0a78d971830393ba106222e0cd3cbb354501 Jul 16 12:47:28.820576 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:28.820544 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" event={"ID":"1c3129e8-fd1f-4d19-9112-1048e92a0645","Type":"ContainerStarted","Data":"7c9e4e9ac65747bd4e6ed1e19368ca1e6c97d1803593b086925d18a645b1bff1"} Jul 16 12:47:28.820979 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:28.820588 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" event={"ID":"1c3129e8-fd1f-4d19-9112-1048e92a0645","Type":"ContainerStarted","Data":"25dd0843032db4a002da5a142a8f0a78d971830393ba106222e0cd3cbb354501"} Jul 16 12:47:28.840977 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:28.840925 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" podStartSLOduration=1.840909229 podStartE2EDuration="1.840909229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:47:28.839393737 +0000 UTC m=+2426.989101172" watchObservedRunningTime="2026-07-16 12:47:28.840909229 +0000 UTC m=+2426.990616681" Jul 16 12:47:37.691670 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:37.691630 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:37.692169 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:37.691710 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:37.704294 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:37.704262 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:47:37.881303 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:47:37.881270 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:48:08.284392 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:08.284354 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj"] Jul 16 12:48:12.199438 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.199400 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk"] Jul 16 12:48:12.203178 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.203152 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.206280 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.206252 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-cf65e4\"/\"kube-root-ca.crt\"" Jul 16 12:48:12.206403 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.206306 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-cf65e4\"/\"default-dockercfg-4z2m4\"" Jul 16 12:48:12.206403 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.206355 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-cf65e4\"/\"openshift-service-ca.crt\"" Jul 16 12:48:12.207527 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.207506 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-cf65e4\"/\"rate-test-model-cf65e4-kserve-self-signed-certs\"" Jul 16 12:48:12.217000 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.216970 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk"] Jul 16 12:48:12.242391 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.242355 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-dshm\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.242576 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.242400 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvdbj\" (UniqueName: \"kubernetes.io/projected/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-kube-api-access-nvdbj\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.242576 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.242426 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-tmp-dir\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.242576 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.242526 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-model-cache\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.242576 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.242562 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-tls-certs\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.242761 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.242614 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-home\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.343999 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.343959 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-dshm\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.343999 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.344000 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvdbj\" (UniqueName: \"kubernetes.io/projected/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-kube-api-access-nvdbj\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.344239 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.344027 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-tmp-dir\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.344239 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.344187 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-model-cache\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.344349 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.344241 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-tls-certs\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.344349 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.344298 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-home\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.344443 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.344421 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-tmp-dir\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.344558 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.344537 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-model-cache\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.344640 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.344615 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-home\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.346419 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.346391 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-dshm\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.346590 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.346573 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-tls-certs\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.353639 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.353611 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvdbj\" (UniqueName: \"kubernetes.io/projected/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-kube-api-access-nvdbj\") pod \"rate-test-model-cf65e4-kserve-69658b6667-772mk\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.514370 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.514329 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:12.672114 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:12.672083 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk"] Jul 16 12:48:12.674022 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:48:12.673990 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda78ef39c_eaad_49c7_8c7a_7f45e9210ca9.slice/crio-ed8392c8b3e1c12779b09d31a257eb8576e8a748129a4a9b2eebcb4a6193afad WatchSource:0}: Error finding container ed8392c8b3e1c12779b09d31a257eb8576e8a748129a4a9b2eebcb4a6193afad: Status 404 returned error can't find the container with id ed8392c8b3e1c12779b09d31a257eb8576e8a748129a4a9b2eebcb4a6193afad Jul 16 12:48:13.060011 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:13.059975 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" event={"ID":"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9","Type":"ContainerStarted","Data":"9f7ce620e2df2558b8c74dfb17a6d74266507b069f974594cfed9f0a33c54cf1"} Jul 16 12:48:13.060011 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:13.060022 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" event={"ID":"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9","Type":"ContainerStarted","Data":"ed8392c8b3e1c12779b09d31a257eb8576e8a748129a4a9b2eebcb4a6193afad"} Jul 16 12:48:13.082916 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:13.082854 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" podStartSLOduration=1.082836217 podStartE2EDuration="1.082836217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:48:13.079551111 +0000 UTC m=+2471.229258545" watchObservedRunningTime="2026-07-16 12:48:13.082836217 +0000 UTC m=+2471.232543659" Jul 16 12:48:22.515474 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:22.515430 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:22.515908 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:22.515615 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:22.528041 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:22.528012 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:23.123408 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.123379 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:48:23.304303 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.304260 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" podUID="1c3129e8-fd1f-4d19-9112-1048e92a0645" containerName="main" containerID="cri-o://7c9e4e9ac65747bd4e6ed1e19368ca1e6c97d1803593b086925d18a645b1bff1" gracePeriod=45 Jul 16 12:48:23.565924 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.565898 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:48:23.658052 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.658011 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf5fs\" (UniqueName: \"kubernetes.io/projected/1c3129e8-fd1f-4d19-9112-1048e92a0645-kube-api-access-rf5fs\") pod \"1c3129e8-fd1f-4d19-9112-1048e92a0645\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " Jul 16 12:48:23.658256 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.658078 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-model-cache\") pod \"1c3129e8-fd1f-4d19-9112-1048e92a0645\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " Jul 16 12:48:23.658256 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.658106 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-dshm\") pod \"1c3129e8-fd1f-4d19-9112-1048e92a0645\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " Jul 16 12:48:23.658256 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.658139 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-home\") pod \"1c3129e8-fd1f-4d19-9112-1048e92a0645\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " Jul 16 12:48:23.658256 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.658171 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-tmp-dir\") pod \"1c3129e8-fd1f-4d19-9112-1048e92a0645\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " Jul 16 12:48:23.658256 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.658227 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3129e8-fd1f-4d19-9112-1048e92a0645-tls-certs\") pod \"1c3129e8-fd1f-4d19-9112-1048e92a0645\" (UID: \"1c3129e8-fd1f-4d19-9112-1048e92a0645\") " Jul 16 12:48:23.658521 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.658442 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-home" (OuterVolumeSpecName: "home") pod "1c3129e8-fd1f-4d19-9112-1048e92a0645" (UID: "1c3129e8-fd1f-4d19-9112-1048e92a0645"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:48:23.658521 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.658458 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c3129e8-fd1f-4d19-9112-1048e92a0645" (UID: "1c3129e8-fd1f-4d19-9112-1048e92a0645"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:48:23.658521 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.658481 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c3129e8-fd1f-4d19-9112-1048e92a0645" (UID: "1c3129e8-fd1f-4d19-9112-1048e92a0645"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:48:23.658819 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.658797 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-model-cache\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:48:23.658874 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.658824 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-home\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:48:23.658874 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.658839 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-tmp-dir\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:48:23.660413 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.660387 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-dshm" (OuterVolumeSpecName: "dshm") pod "1c3129e8-fd1f-4d19-9112-1048e92a0645" (UID: "1c3129e8-fd1f-4d19-9112-1048e92a0645"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:48:23.660782 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.660755 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3129e8-fd1f-4d19-9112-1048e92a0645-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c3129e8-fd1f-4d19-9112-1048e92a0645" (UID: "1c3129e8-fd1f-4d19-9112-1048e92a0645"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:48:23.660912 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.660895 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3129e8-fd1f-4d19-9112-1048e92a0645-kube-api-access-rf5fs" (OuterVolumeSpecName: "kube-api-access-rf5fs") pod "1c3129e8-fd1f-4d19-9112-1048e92a0645" (UID: "1c3129e8-fd1f-4d19-9112-1048e92a0645"). InnerVolumeSpecName "kube-api-access-rf5fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:48:23.759911 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.759872 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf5fs\" (UniqueName: \"kubernetes.io/projected/1c3129e8-fd1f-4d19-9112-1048e92a0645-kube-api-access-rf5fs\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:48:23.759911 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.759907 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3129e8-fd1f-4d19-9112-1048e92a0645-dshm\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:48:23.760122 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:23.759920 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3129e8-fd1f-4d19-9112-1048e92a0645-tls-certs\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:48:24.118772 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:24.118664 2559 generic.go:356] "Generic (PLEG): container finished" podID="1c3129e8-fd1f-4d19-9112-1048e92a0645" containerID="7c9e4e9ac65747bd4e6ed1e19368ca1e6c97d1803593b086925d18a645b1bff1" exitCode=0 Jul 16 12:48:24.118772 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:24.118718 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" event={"ID":"1c3129e8-fd1f-4d19-9112-1048e92a0645","Type":"ContainerDied","Data":"7c9e4e9ac65747bd4e6ed1e19368ca1e6c97d1803593b086925d18a645b1bff1"} Jul 16 12:48:24.118772 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:24.118763 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" event={"ID":"1c3129e8-fd1f-4d19-9112-1048e92a0645","Type":"ContainerDied","Data":"25dd0843032db4a002da5a142a8f0a78d971830393ba106222e0cd3cbb354501"} Jul 16 12:48:24.119008 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:24.118784 2559 scope.go:117] "RemoveContainer" containerID="7c9e4e9ac65747bd4e6ed1e19368ca1e6c97d1803593b086925d18a645b1bff1" Jul 16 12:48:24.119008 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:24.118730 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj" Jul 16 12:48:24.128269 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:24.128248 2559 scope.go:117] "RemoveContainer" containerID="7c9e4e9ac65747bd4e6ed1e19368ca1e6c97d1803593b086925d18a645b1bff1" Jul 16 12:48:24.128533 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:48:24.128518 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9e4e9ac65747bd4e6ed1e19368ca1e6c97d1803593b086925d18a645b1bff1\": container with ID starting with 7c9e4e9ac65747bd4e6ed1e19368ca1e6c97d1803593b086925d18a645b1bff1 not found: ID does not exist" containerID="7c9e4e9ac65747bd4e6ed1e19368ca1e6c97d1803593b086925d18a645b1bff1" Jul 16 12:48:24.128577 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:24.128542 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9e4e9ac65747bd4e6ed1e19368ca1e6c97d1803593b086925d18a645b1bff1"} err="failed to get container status \"7c9e4e9ac65747bd4e6ed1e19368ca1e6c97d1803593b086925d18a645b1bff1\": rpc error: code = NotFound desc = could not find container \"7c9e4e9ac65747bd4e6ed1e19368ca1e6c97d1803593b086925d18a645b1bff1\": container with ID starting with 7c9e4e9ac65747bd4e6ed1e19368ca1e6c97d1803593b086925d18a645b1bff1 not found: ID does not exist" Jul 16 12:48:24.145020 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:24.144995 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj"] Jul 16 12:48:24.148502 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:24.148479 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/sub-test-model-3c72a3-kserve-5f47f6df5c-llsjj"] Jul 16 12:48:24.557343 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:48:24.557307 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c3129e8-fd1f-4d19-9112-1048e92a0645" path="/var/lib/kubelet/pods/1c3129e8-fd1f-4d19-9112-1048e92a0645/volumes" Jul 16 12:49:34.644731 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:34.644690 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk"] Jul 16 12:49:49.667065 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:49.666997 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" podUID="a78ef39c-eaad-49c7-8c7a-7f45e9210ca9" containerName="main" containerID="cri-o://9f7ce620e2df2558b8c74dfb17a6d74266507b069f974594cfed9f0a33c54cf1" gracePeriod=45 Jul 16 12:49:49.920870 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:49.920806 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:49:49.924916 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:49.924894 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-dshm\") pod \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " Jul 16 12:49:49.925059 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:49.924925 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-tls-certs\") pod \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " Jul 16 12:49:49.925059 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:49.924962 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-tmp-dir\") pod \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " Jul 16 12:49:49.925059 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:49.925018 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-model-cache\") pod \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " Jul 16 12:49:49.925059 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:49.925050 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-home\") pod \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " Jul 16 12:49:49.925284 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:49.925107 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvdbj\" (UniqueName: \"kubernetes.io/projected/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-kube-api-access-nvdbj\") pod \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\" (UID: \"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9\") " Jul 16 12:49:49.925339 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:49.925275 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a78ef39c-eaad-49c7-8c7a-7f45e9210ca9" (UID: "a78ef39c-eaad-49c7-8c7a-7f45e9210ca9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:49:49.925339 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:49.925294 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-model-cache" (OuterVolumeSpecName: "model-cache") pod "a78ef39c-eaad-49c7-8c7a-7f45e9210ca9" (UID: "a78ef39c-eaad-49c7-8c7a-7f45e9210ca9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:49:49.925448 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:49.925417 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-home" (OuterVolumeSpecName: "home") pod "a78ef39c-eaad-49c7-8c7a-7f45e9210ca9" (UID: "a78ef39c-eaad-49c7-8c7a-7f45e9210ca9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:49:49.925592 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:49.925557 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-tmp-dir\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:49:49.925656 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:49.925596 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-model-cache\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:49:49.925656 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:49.925611 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-home\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:49:49.927172 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:49.927151 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-dshm" (OuterVolumeSpecName: "dshm") pod "a78ef39c-eaad-49c7-8c7a-7f45e9210ca9" (UID: "a78ef39c-eaad-49c7-8c7a-7f45e9210ca9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:49:49.927253 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:49.927175 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a78ef39c-eaad-49c7-8c7a-7f45e9210ca9" (UID: "a78ef39c-eaad-49c7-8c7a-7f45e9210ca9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:49:49.927253 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:49.927219 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-kube-api-access-nvdbj" (OuterVolumeSpecName: "kube-api-access-nvdbj") pod "a78ef39c-eaad-49c7-8c7a-7f45e9210ca9" (UID: "a78ef39c-eaad-49c7-8c7a-7f45e9210ca9"). InnerVolumeSpecName "kube-api-access-nvdbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:49:50.026531 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:50.026487 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-dshm\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:49:50.026531 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:50.026533 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-tls-certs\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:49:50.026782 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:50.026550 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvdbj\" (UniqueName: \"kubernetes.io/projected/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9-kube-api-access-nvdbj\") on node \"ip-10-0-143-101.ec2.internal\" DevicePath \"\"" Jul 16 12:49:50.576154 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:50.576118 2559 generic.go:356] "Generic (PLEG): container finished" podID="a78ef39c-eaad-49c7-8c7a-7f45e9210ca9" containerID="9f7ce620e2df2558b8c74dfb17a6d74266507b069f974594cfed9f0a33c54cf1" exitCode=0 Jul 16 12:49:50.576339 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:50.576166 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" event={"ID":"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9","Type":"ContainerDied","Data":"9f7ce620e2df2558b8c74dfb17a6d74266507b069f974594cfed9f0a33c54cf1"} Jul 16 12:49:50.576339 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:50.576178 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" Jul 16 12:49:50.576339 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:50.576198 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk" event={"ID":"a78ef39c-eaad-49c7-8c7a-7f45e9210ca9","Type":"ContainerDied","Data":"ed8392c8b3e1c12779b09d31a257eb8576e8a748129a4a9b2eebcb4a6193afad"} Jul 16 12:49:50.576339 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:50.576213 2559 scope.go:117] "RemoveContainer" containerID="9f7ce620e2df2558b8c74dfb17a6d74266507b069f974594cfed9f0a33c54cf1" Jul 16 12:49:50.585406 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:50.585387 2559 scope.go:117] "RemoveContainer" containerID="9f7ce620e2df2558b8c74dfb17a6d74266507b069f974594cfed9f0a33c54cf1" Jul 16 12:49:50.585702 ip-10-0-143-101 kubenswrapper[2559]: E0716 12:49:50.585666 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f7ce620e2df2558b8c74dfb17a6d74266507b069f974594cfed9f0a33c54cf1\": container with ID starting with 9f7ce620e2df2558b8c74dfb17a6d74266507b069f974594cfed9f0a33c54cf1 not found: ID does not exist" containerID="9f7ce620e2df2558b8c74dfb17a6d74266507b069f974594cfed9f0a33c54cf1" Jul 16 12:49:50.585759 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:50.585711 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f7ce620e2df2558b8c74dfb17a6d74266507b069f974594cfed9f0a33c54cf1"} err="failed to get container status \"9f7ce620e2df2558b8c74dfb17a6d74266507b069f974594cfed9f0a33c54cf1\": rpc error: code = NotFound desc = could not find container \"9f7ce620e2df2558b8c74dfb17a6d74266507b069f974594cfed9f0a33c54cf1\": container with ID starting with 9f7ce620e2df2558b8c74dfb17a6d74266507b069f974594cfed9f0a33c54cf1 not found: ID does not exist" Jul 16 12:49:50.596259 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:50.596230 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk"] Jul 16 12:49:50.611494 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:50.611455 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/rate-test-model-cf65e4-kserve-69658b6667-772mk"] Jul 16 12:49:52.557192 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:49:52.557156 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a78ef39c-eaad-49c7-8c7a-7f45e9210ca9" path="/var/lib/kubelet/pods/a78ef39c-eaad-49c7-8c7a-7f45e9210ca9/volumes" Jul 16 12:50:10.098429 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:10.098393 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5dc7d676c7-h2t97_003e4c33-cbd4-4fab-8fe8-d4c9f418ddb0/authorino/0.log" Jul 16 12:50:16.941350 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:16.941318 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc9bf88fc-54dx9_7e3017a4-d440-47ff-b312-89920b129880/copy-manifests/0.log" Jul 16 12:50:16.949359 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:16.949335 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc9bf88fc-54dx9_7e3017a4-d440-47ff-b312-89920b129880/manager/0.log" Jul 16 12:50:17.914321 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:17.914292 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-api-b-47e820-778f585dff-44r9p_544647d1-97b6-4f1f-8856-7afc0d35d1b9/maas-api/0.log" Jul 16 12:50:18.165116 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:18.165030 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-a-28d017-64948b66f7-59svn_62b5a0fe-3d54-409c-8581-2c85bcf83923/maas-api/0.log" Jul 16 12:50:18.299611 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:18.299583 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-b-9ac0a7-555c99fbc7-smxm5_f1a02f00-ddb3-4fa6-9193-4b27a2df8e19/maas-api/0.log" Jul 16 12:50:18.424609 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:18.424533 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-migrate-051a7b7b-76bdd5cdb8-h9wv4_e8980583-2084-44f9-a54d-ef294253fd9a/maas-api/0.log" Jul 16 12:50:18.548347 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:18.548311 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-mt-df8d3cf1-6c9d7f676d-7tgxk_edf93aa4-bb5c-4d56-9e7d-cfb1149f13d1/maas-api/0.log" Jul 16 12:50:18.923040 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:18.923005 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29736750-cnsnd_56dc5ea0-b5dc-49d9-bd97-34ad199b3acc/cleanup/0.log" Jul 16 12:50:19.048226 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:19.048199 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29736765-fz7ts_49a13dea-5090-48ee-9dba-4fc4c28a3d76/cleanup/0.log" Jul 16 12:50:19.792346 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:19.792318 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-inf-b-9ac0a7-29736765-bxt5n_88bfe16a-308c-459b-a870-7da4b6437df7/cleanup/0.log" Jul 16 12:50:21.857872 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:21.857838 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5dc7d676c7-h2t97_003e4c33-cbd4-4fab-8fe8-d4c9f418ddb0/authorino/0.log" Jul 16 12:50:22.531213 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:22.531179 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-spmzx_c222bb8c-3d5f-4ed6-84de-936443eaba9b/manager/0.log" Jul 16 12:50:23.480571 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:23.480542 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-569f4f6548-cpsgs_c97c87ff-85f9-4b52-ab56-16d7dff6c46c/kube-auth-proxy/0.log" Jul 16 12:50:23.858164 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:23.858126 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-mbxwh_08da884b-cd39-43ed-bab1-62c7dfce73e8/payload-pre-processing/0.log" Jul 16 12:50:24.483091 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:24.483018 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-tc8cr_95e9b458-6e82-408d-a3a6-ee1c1fa42b19/main/0.log" Jul 16 12:50:24.867738 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:24.867705 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbd86td_22b5d613-abe9-4d27-b954-e886cec96f01/main/0.log" Jul 16 12:50:25.113584 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:25.113554 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-hbzds_f56f4e7d-b360-4313-8455-23d20f3c3b70/main/0.log" Jul 16 12:50:34.595128 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:34.595095 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mbwtf_6e31220b-1279-4b66-87c2-ea2948393fd1/global-pull-secret-syncer/0.log" Jul 16 12:50:34.768264 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:34.768232 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f4btg_560b077f-9bc1-4c85-a34a-af83df0daed5/konnectivity-agent/0.log" Jul 16 12:50:34.847417 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:34.847337 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-101.ec2.internal_3b7f01a0953299e98dac4176c400f9a5/haproxy/0.log" Jul 16 12:50:39.804123 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:39.804087 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5dc7d676c7-h2t97_003e4c33-cbd4-4fab-8fe8-d4c9f418ddb0/authorino/0.log" Jul 16 12:50:40.101549 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:40.101510 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-spmzx_c222bb8c-3d5f-4ed6-84de-936443eaba9b/manager/0.log" Jul 16 12:50:42.081713 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.081657 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-8rx2b_b8cae4b2-a2c7-420d-832e-37982ffe2678/cluster-monitoring-operator/0.log" Jul 16 12:50:42.111278 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.111250 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-f2hjc_9ac5634c-d4f8-4e17-a719-6a6dafff2d5a/kube-state-metrics/0.log" Jul 16 12:50:42.134671 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.134643 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-f2hjc_9ac5634c-d4f8-4e17-a719-6a6dafff2d5a/kube-rbac-proxy-main/0.log" Jul 16 12:50:42.159721 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.159669 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-f2hjc_9ac5634c-d4f8-4e17-a719-6a6dafff2d5a/kube-rbac-proxy-self/0.log" Jul 16 12:50:42.266151 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.266113 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c2r95_42b9dc92-8c58-474e-aa83-806dee5ed2df/node-exporter/0.log" Jul 16 12:50:42.288355 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.288326 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c2r95_42b9dc92-8c58-474e-aa83-806dee5ed2df/kube-rbac-proxy/0.log" Jul 16 12:50:42.320051 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.320023 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c2r95_42b9dc92-8c58-474e-aa83-806dee5ed2df/init-textfile/0.log" Jul 16 12:50:42.531323 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.531295 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-wsr5g_231fbb85-1b53-4342-99bc-a25f64972033/kube-rbac-proxy-main/0.log" Jul 16 12:50:42.554993 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.554967 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-wsr5g_231fbb85-1b53-4342-99bc-a25f64972033/kube-rbac-proxy-self/0.log" Jul 16 12:50:42.591368 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.591340 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-wsr5g_231fbb85-1b53-4342-99bc-a25f64972033/openshift-state-metrics/0.log" Jul 16 12:50:42.627785 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.627751 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7476a2bc-798c-445b-ba4f-f463fca083a3/prometheus/0.log" Jul 16 12:50:42.653632 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.653603 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7476a2bc-798c-445b-ba4f-f463fca083a3/config-reloader/0.log" Jul 16 12:50:42.677879 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.677850 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7476a2bc-798c-445b-ba4f-f463fca083a3/thanos-sidecar/0.log" Jul 16 12:50:42.701040 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.701014 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7476a2bc-798c-445b-ba4f-f463fca083a3/kube-rbac-proxy-web/0.log" Jul 16 12:50:42.725377 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.725351 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7476a2bc-798c-445b-ba4f-f463fca083a3/kube-rbac-proxy/0.log" Jul 16 12:50:42.748351 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.748321 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7476a2bc-798c-445b-ba4f-f463fca083a3/kube-rbac-proxy-thanos/0.log" Jul 16 12:50:42.791923 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.791796 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7476a2bc-798c-445b-ba4f-f463fca083a3/init-config-reloader/0.log" Jul 16 12:50:42.900866 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.900837 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d5656d965-k22f9_b8049df4-0e4e-4afd-9888-b45d70b596a1/telemeter-client/0.log" Jul 16 12:50:42.926262 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.926227 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d5656d965-k22f9_b8049df4-0e4e-4afd-9888-b45d70b596a1/reload/0.log" Jul 16 12:50:42.950065 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:42.950035 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d5656d965-k22f9_b8049df4-0e4e-4afd-9888-b45d70b596a1/kube-rbac-proxy/0.log" Jul 16 12:50:44.155597 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.155561 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6"] Jul 16 12:50:44.156042 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.156026 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a78ef39c-eaad-49c7-8c7a-7f45e9210ca9" containerName="main" Jul 16 12:50:44.156103 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.156043 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a78ef39c-eaad-49c7-8c7a-7f45e9210ca9" containerName="main" Jul 16 12:50:44.156103 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.156056 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c3129e8-fd1f-4d19-9112-1048e92a0645" containerName="main" Jul 16 12:50:44.156103 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.156061 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c3129e8-fd1f-4d19-9112-1048e92a0645" containerName="main" Jul 16 12:50:44.156223 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.156140 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a78ef39c-eaad-49c7-8c7a-7f45e9210ca9" containerName="main" Jul 16 12:50:44.156223 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.156150 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c3129e8-fd1f-4d19-9112-1048e92a0645" containerName="main" Jul 16 12:50:44.159613 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.159595 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.162127 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.162106 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xm7nj\"/\"openshift-service-ca.crt\"" Jul 16 12:50:44.162247 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.162145 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xm7nj\"/\"kube-root-ca.crt\"" Jul 16 12:50:44.163070 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.163052 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xm7nj\"/\"default-dockercfg-7f7h2\"" Jul 16 12:50:44.168614 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.168586 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6"] Jul 16 12:50:44.228275 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.228240 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07956150-be26-437f-8049-cc2b616fbcb4-lib-modules\") pod \"perf-node-gather-daemonset-v42l6\" (UID: \"07956150-be26-437f-8049-cc2b616fbcb4\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.228275 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.228280 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07956150-be26-437f-8049-cc2b616fbcb4-proc\") pod \"perf-node-gather-daemonset-v42l6\" (UID: \"07956150-be26-437f-8049-cc2b616fbcb4\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.228518 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.228368 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tv2p\" (UniqueName: \"kubernetes.io/projected/07956150-be26-437f-8049-cc2b616fbcb4-kube-api-access-2tv2p\") pod \"perf-node-gather-daemonset-v42l6\" (UID: \"07956150-be26-437f-8049-cc2b616fbcb4\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.228518 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.228452 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07956150-be26-437f-8049-cc2b616fbcb4-sys\") pod \"perf-node-gather-daemonset-v42l6\" (UID: \"07956150-be26-437f-8049-cc2b616fbcb4\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.228607 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.228524 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07956150-be26-437f-8049-cc2b616fbcb4-podres\") pod \"perf-node-gather-daemonset-v42l6\" (UID: \"07956150-be26-437f-8049-cc2b616fbcb4\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.309815 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.309781 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-mx7qx_3a2c5faa-25b2-4e17-852d-56aaf8403d1f/networking-console-plugin/0.log" Jul 16 12:50:44.329835 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.329799 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tv2p\" (UniqueName: \"kubernetes.io/projected/07956150-be26-437f-8049-cc2b616fbcb4-kube-api-access-2tv2p\") pod \"perf-node-gather-daemonset-v42l6\" (UID: \"07956150-be26-437f-8049-cc2b616fbcb4\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.330014 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.329853 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07956150-be26-437f-8049-cc2b616fbcb4-sys\") pod \"perf-node-gather-daemonset-v42l6\" (UID: \"07956150-be26-437f-8049-cc2b616fbcb4\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.330014 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.329940 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07956150-be26-437f-8049-cc2b616fbcb4-podres\") pod \"perf-node-gather-daemonset-v42l6\" (UID: \"07956150-be26-437f-8049-cc2b616fbcb4\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.330014 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.329983 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07956150-be26-437f-8049-cc2b616fbcb4-lib-modules\") pod \"perf-node-gather-daemonset-v42l6\" (UID: \"07956150-be26-437f-8049-cc2b616fbcb4\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.330014 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.329996 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07956150-be26-437f-8049-cc2b616fbcb4-sys\") pod \"perf-node-gather-daemonset-v42l6\" (UID: \"07956150-be26-437f-8049-cc2b616fbcb4\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.330168 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.330014 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07956150-be26-437f-8049-cc2b616fbcb4-proc\") pod \"perf-node-gather-daemonset-v42l6\" (UID: \"07956150-be26-437f-8049-cc2b616fbcb4\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.330168 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.330065 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07956150-be26-437f-8049-cc2b616fbcb4-proc\") pod \"perf-node-gather-daemonset-v42l6\" (UID: \"07956150-be26-437f-8049-cc2b616fbcb4\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.330168 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.330108 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07956150-be26-437f-8049-cc2b616fbcb4-podres\") pod \"perf-node-gather-daemonset-v42l6\" (UID: \"07956150-be26-437f-8049-cc2b616fbcb4\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.330168 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.330132 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07956150-be26-437f-8049-cc2b616fbcb4-lib-modules\") pod \"perf-node-gather-daemonset-v42l6\" (UID: \"07956150-be26-437f-8049-cc2b616fbcb4\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.339506 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.339472 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tv2p\" (UniqueName: \"kubernetes.io/projected/07956150-be26-437f-8049-cc2b616fbcb4-kube-api-access-2tv2p\") pod \"perf-node-gather-daemonset-v42l6\" (UID: \"07956150-be26-437f-8049-cc2b616fbcb4\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.471395 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.471307 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.617872 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.617845 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6"] Jul 16 12:50:44.619456 ip-10-0-143-101 kubenswrapper[2559]: W0716 12:50:44.619426 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod07956150_be26_437f_8049_cc2b616fbcb4.slice/crio-10794d63e2785a1260e479e07ebdd1e148ac1e39f4bb2852add67739985f761b WatchSource:0}: Error finding container 10794d63e2785a1260e479e07ebdd1e148ac1e39f4bb2852add67739985f761b: Status 404 returned error can't find the container with id 10794d63e2785a1260e479e07ebdd1e148ac1e39f4bb2852add67739985f761b Jul 16 12:50:44.857730 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.857687 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" event={"ID":"07956150-be26-437f-8049-cc2b616fbcb4","Type":"ContainerStarted","Data":"69b98b0676c07714d9ac742855b7825ae521dab4a91f62cce4088f6d2b857e79"} Jul 16 12:50:44.857919 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.857746 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" event={"ID":"07956150-be26-437f-8049-cc2b616fbcb4","Type":"ContainerStarted","Data":"10794d63e2785a1260e479e07ebdd1e148ac1e39f4bb2852add67739985f761b"} Jul 16 12:50:44.857919 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.857863 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:44.877987 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:44.877934 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" podStartSLOduration=0.87791608 podStartE2EDuration="877.91608ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:50:44.875640282 +0000 UTC m=+2623.025347714" watchObservedRunningTime="2026-07-16 12:50:44.87791608 +0000 UTC m=+2623.027623513" Jul 16 12:50:45.367871 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:45.367849 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-85d67c5cb5-j46rk_a5b88bcc-733a-4ac9-9a2a-691b09afab1a/console/0.log" Jul 16 12:50:47.106882 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:47.106844 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f55c4_268f88f9-d545-4d19-9bc8-8695df1dc92c/dns/0.log" Jul 16 12:50:47.149892 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:47.149840 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f55c4_268f88f9-d545-4d19-9bc8-8695df1dc92c/kube-rbac-proxy/0.log" Jul 16 12:50:47.408972 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:47.408883 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6vr5g_d1fe16f9-e90b-4dbf-8462-92f01545a15e/dns-node-resolver/0.log" Jul 16 12:50:48.123486 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:48.123456 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6wxl5_d4b17153-bc06-4588-a2ba-6a11d0de2da9/node-ca/0.log" Jul 16 12:50:49.326948 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:49.326924 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-569f4f6548-cpsgs_c97c87ff-85f9-4b52-ab56-16d7dff6c46c/kube-auth-proxy/0.log" Jul 16 12:50:49.476376 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:49.476348 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-mbxwh_08da884b-cd39-43ed-bab1-62c7dfce73e8/payload-pre-processing/0.log" Jul 16 12:50:50.329313 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:50.329283 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r4f95_94335d8a-acdd-4739-8741-ce365e66d8c5/serve-healthcheck-canary/0.log" Jul 16 12:50:50.872950 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:50.872922 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-v42l6" Jul 16 12:50:50.907596 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:50.907559 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qlcvh_8a0c36a7-ae21-445b-863b-bab16f7d69b7/kube-rbac-proxy/0.log" Jul 16 12:50:50.931506 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:50.931473 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qlcvh_8a0c36a7-ae21-445b-863b-bab16f7d69b7/exporter/0.log" Jul 16 12:50:50.954974 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:50.954941 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qlcvh_8a0c36a7-ae21-445b-863b-bab16f7d69b7/extractor/0.log" Jul 16 12:50:53.881453 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:53.881422 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc9bf88fc-54dx9_7e3017a4-d440-47ff-b312-89920b129880/manager/0.log" Jul 16 12:50:53.905218 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:53.905195 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc9bf88fc-54dx9_7e3017a4-d440-47ff-b312-89920b129880/copy-manifests/0.log" Jul 16 12:50:54.533610 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:54.533580 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-569f4f6548-cpsgs_c97c87ff-85f9-4b52-ab56-16d7dff6c46c/kube-auth-proxy/0.log" Jul 16 12:50:54.617107 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:54.617074 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-mbxwh_08da884b-cd39-43ed-bab1-62c7dfce73e8/payload-pre-processing/0.log" Jul 16 12:50:55.299734 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:55.299702 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-69c8d477dd-sdlvw_7319953c-b07a-41ad-a321-141905dce49a/manager/0.log" Jul 16 12:50:56.022773 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:56.022744 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-569f4f6548-cpsgs_c97c87ff-85f9-4b52-ab56-16d7dff6c46c/kube-auth-proxy/0.log" Jul 16 12:50:56.185486 ip-10-0-143-101 kubenswrapper[2559]: I0716 12:50:56.185452 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-mbxwh_08da884b-cd39-43ed-bab1-62c7dfce73e8/payload-pre-processing/0.log"