Jul 12 18:17:17.979137 ip-10-0-132-5 systemd[1]: Starting Kubernetes Kubelet... Jul 12 18:17:18.710668 ip-10-0-132-5 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 18:17:18.710668 ip-10-0-132-5 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 12 18:17:18.710668 ip-10-0-132-5 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 18:17:18.710668 ip-10-0-132-5 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 12 18:17:18.710668 ip-10-0-132-5 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 18:17:18.710668 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.530159 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 12 18:17:18.710668 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536176 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 18:17:18.710668 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536188 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 18:17:18.710668 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536193 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 18:17:18.710668 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536196 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 18:17:18.710668 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536199 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 18:17:18.710668 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536202 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 18:17:18.710668 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536205 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 18:17:18.710668 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536208 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 18:17:18.613051 ip-10-0-132-5 systemd[1]: Started Kubernetes Kubelet. Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536212 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536215 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536218 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536221 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536224 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536227 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536230 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536233 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536237 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536247 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536250 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536253 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536256 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536260 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536265 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536269 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536272 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536275 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 18:17:18.711703 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536278 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536281 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536283 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536286 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536291 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536297 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536300 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536305 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536308 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536312 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536315 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536319 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536322 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536325 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536328 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536331 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536334 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536337 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536340 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 18:17:18.713218 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536343 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536346 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536349 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536351 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536354 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536357 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536360 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536362 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536365 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536368 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536371 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536374 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536377 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536379 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536382 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536385 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536387 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536390 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536393 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536395 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 18:17:18.988323 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536398 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536401 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536404 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536407 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536410 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536414 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536417 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536421 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536424 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536427 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536430 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536433 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536437 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536440 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536442 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536445 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536448 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536452 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536454 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536459 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 18:17:18.989138 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536463 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536466 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536469 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536472 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536474 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536477 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536480 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536483 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536485 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536488 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536491 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536494 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536496 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.536499 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537320 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537329 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537333 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537336 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537339 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537345 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537348 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 18:17:18.989813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537351 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537354 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537357 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537361 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537364 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537367 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537370 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537374 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537377 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537381 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537386 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537389 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537392 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537395 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537398 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537401 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537404 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537407 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537409 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537412 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 18:17:18.990561 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537416 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537419 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537422 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537425 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537427 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537431 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537434 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537437 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537440 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537443 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537446 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537449 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537452 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537455 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537457 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537461 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537464 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537469 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537472 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537475 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 18:17:18.991290 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537478 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537481 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537484 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537488 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537494 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537497 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537500 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537505 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537508 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537511 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537514 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537517 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537520 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537523 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537525 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537528 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537531 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537534 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537537 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537540 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 18:17:18.991938 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537542 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537545 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537548 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537551 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537554 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537557 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537560 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537563 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537567 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537577 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537580 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537583 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537586 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537589 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537592 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537595 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537598 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537601 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537604 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 18:17:18.992814 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537608 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537611 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537614 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537617 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537620 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537622 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537625 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537628 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537631 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537634 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537637 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537639 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.537642 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538509 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538517 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538521 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538525 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538529 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538533 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538536 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 18:17:18.993471 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538539 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538542 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538546 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538550 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538552 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538555 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538558 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538561 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538564 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538567 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538570 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538572 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538578 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538581 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538584 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538587 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538590 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538592 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538595 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538598 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 12 18:17:18.994165 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538601 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538604 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538606 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538609 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538612 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538615 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538619 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538623 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538626 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538629 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538633 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538637 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538640 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538643 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538649 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538652 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538654 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538658 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538661 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 18:17:18.994790 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538665 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538668 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538671 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538673 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538676 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538680 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538683 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538686 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538689 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538692 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538695 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538697 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538700 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538703 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538706 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538708 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538711 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538715 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538717 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538720 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 18:17:18.995450 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538723 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538726 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538729 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538732 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538737 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538740 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538744 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538748 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538751 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538754 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538758 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538761 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538763 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538767 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538769 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538772 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538775 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538779 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538782 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 18:17:18.996091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538785 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538788 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538791 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538793 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538796 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538798 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538801 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538804 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538807 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538810 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538813 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538815 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538818 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.538821 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.538920 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.538931 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.538941 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.538951 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.538958 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.538963 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.538970 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 12 18:17:18.996689 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.538974 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.538978 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.538981 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.538985 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.538988 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.538992 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.538995 2576 flags.go:64] FLAG: --cgroup-root="" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.538998 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539001 2576 flags.go:64] FLAG: --client-ca-file="" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539006 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539009 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539014 2576 flags.go:64] FLAG: --cluster-domain="" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539017 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539020 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539023 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539027 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539032 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539035 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539039 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539042 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539046 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539049 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539052 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539055 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 12 18:17:18.997306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539058 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539063 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539066 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539069 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539072 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539075 2576 flags.go:64] FLAG: --enable-server="true" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539079 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539086 2576 flags.go:64] FLAG: --event-burst="100" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539090 2576 flags.go:64] FLAG: --event-qps="50" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539094 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539097 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539100 2576 flags.go:64] FLAG: --eviction-hard="" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539104 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539108 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539111 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539114 2576 flags.go:64] FLAG: --eviction-soft="" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539117 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539123 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539126 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539129 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539132 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539136 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539139 2576 flags.go:64] FLAG: --feature-gates="" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539145 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539150 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 12 18:17:18.998152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539155 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539158 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539161 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539165 2576 flags.go:64] FLAG: --help="false" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539168 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-132-5.ec2.internal" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539171 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539175 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539178 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539181 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539185 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539188 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539191 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539194 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539197 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539206 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539210 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539213 2576 flags.go:64] FLAG: --kube-reserved="" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539217 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539220 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539223 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539226 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539229 2576 flags.go:64] FLAG: --lock-file="" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539232 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539235 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 12 18:17:18.998997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539241 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539246 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539249 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539252 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539255 2576 flags.go:64] FLAG: --logging-format="text" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539258 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539262 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539265 2576 flags.go:64] FLAG: --manifest-url="" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539268 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539273 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539276 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539281 2576 flags.go:64] FLAG: --max-pods="110" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539284 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539287 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539290 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539294 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539297 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539300 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539303 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539311 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539314 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539318 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539323 2576 flags.go:64] FLAG: --pod-cidr="" Jul 12 18:17:18.999714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539327 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539333 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539337 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539341 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539344 2576 flags.go:64] FLAG: --port="10250" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539347 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539350 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f8faa10d9545d0d9" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539354 2576 flags.go:64] FLAG: --qos-reserved="" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539357 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539360 2576 flags.go:64] FLAG: --register-node="true" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539365 2576 flags.go:64] FLAG: --register-with-taints="" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539371 2576 flags.go:64] FLAG: --registry-burst="10" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539375 2576 flags.go:64] FLAG: --registry-qps="5" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539378 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539381 2576 flags.go:64] FLAG: --reserved-memory="" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539385 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539388 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539391 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539395 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539398 2576 flags.go:64] FLAG: --runonce="false" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539401 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539404 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539407 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539410 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539414 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539417 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 12 18:17:19.000393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539420 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539423 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539426 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539430 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539433 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539437 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539441 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539445 2576 flags.go:64] FLAG: --system-cgroups="" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539448 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539454 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539457 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539460 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539464 2576 flags.go:64] FLAG: --tls-min-version="" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539467 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539470 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539473 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539478 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539481 2576 flags.go:64] FLAG: --v="2" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539485 2576 flags.go:64] FLAG: --version="false" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539489 2576 flags.go:64] FLAG: --vmodule="" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539493 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.539497 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539650 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539656 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539659 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 12 18:17:19.001164 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539663 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539666 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539669 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539672 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539675 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539678 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539681 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539684 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539686 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539689 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539692 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539695 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539699 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539702 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539705 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539708 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539711 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539715 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539718 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539720 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 18:17:19.004648 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539723 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539726 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539729 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539733 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539736 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539739 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539742 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539745 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539748 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539750 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539753 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539756 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539758 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539761 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539764 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539767 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539770 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539772 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539775 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539778 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 18:17:19.005573 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539781 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539784 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539786 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539789 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539793 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539796 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539813 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539818 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539822 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539826 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539829 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539832 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539835 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539838 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539841 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539846 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539850 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539853 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539856 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 18:17:19.253706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539859 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539862 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539865 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539868 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539871 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539873 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539876 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539879 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539882 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539885 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539888 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539891 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539895 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539898 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539902 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539905 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539908 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539912 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539915 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 18:17:19.254813 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539918 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 18:17:19.255734 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539921 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 18:17:19.255734 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539924 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 18:17:19.255734 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539927 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 18:17:19.255734 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539930 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 18:17:19.255734 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539933 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 18:17:19.255734 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539935 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 18:17:19.255734 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539938 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 18:17:19.255734 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539941 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 18:17:19.255734 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539945 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 18:17:19.255734 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539948 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 18:17:19.255734 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539950 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 18:17:19.255734 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539953 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 18:17:19.255734 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539956 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 18:17:19.255734 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539959 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 18:17:19.255734 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539961 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 18:17:19.255734 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539964 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 18:17:19.255734 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.539967 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 18:17:19.256658 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.540629 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 18:17:19.256658 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.548209 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 12 18:17:19.256658 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.548226 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 12 18:17:19.256658 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.548248 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 12 18:17:19.256658 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.548256 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 12 18:17:19.256658 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548327 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 18:17:19.256658 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548333 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 18:17:19.256658 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548336 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 18:17:19.256658 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548340 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 18:17:19.256658 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548343 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 18:17:19.256658 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548346 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 18:17:19.256658 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548350 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 18:17:19.256658 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548353 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 18:17:19.256658 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548356 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 18:17:19.256658 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548359 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548362 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548365 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548368 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548371 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548374 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548377 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548382 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548386 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548390 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548393 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548396 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548400 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548403 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548407 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548410 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548413 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548418 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548422 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 18:17:19.257575 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548425 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548428 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548431 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548434 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548438 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548440 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548443 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548446 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548449 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548454 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548457 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548460 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548463 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548466 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548469 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548472 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548475 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548478 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548480 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548483 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 18:17:19.258512 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548486 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548489 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548492 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548495 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548498 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548501 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548504 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548507 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548509 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548512 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548518 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548521 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548524 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548527 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548532 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548536 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548539 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548542 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548546 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 18:17:19.259436 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548549 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548552 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548557 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548560 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548563 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548566 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548569 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548572 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548575 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548578 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548581 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548584 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548586 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548590 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548594 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548598 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548601 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548604 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548607 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 18:17:19.261210 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548610 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 18:17:19.262223 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548613 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 18:17:19.262223 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548615 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 18:17:19.262223 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548618 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 18:17:19.262223 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548623 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 18:17:19.262223 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548626 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 18:17:19.262223 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548629 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 18:17:19.262223 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548632 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 18:17:19.262223 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548635 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 18:17:19.262223 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548639 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 18:17:19.262223 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548642 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 18:17:19.262223 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548645 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 18:17:19.262223 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548647 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 18:17:19.262223 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.548653 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 18:17:19.262223 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548795 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 18:17:19.262223 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548802 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548805 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548809 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548812 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548815 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548818 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548820 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548823 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548826 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548829 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548832 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548834 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548837 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548840 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548843 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548846 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548848 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548851 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548854 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548872 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 18:17:19.263148 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548877 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548880 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548884 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548887 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548890 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548893 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548896 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548899 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548902 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548905 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548908 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548910 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548914 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548917 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548920 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548923 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548926 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548928 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548931 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548934 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 18:17:19.264079 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548937 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548939 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548942 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548945 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548948 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548951 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548954 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548957 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548959 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548962 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548965 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548967 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548970 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548973 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548979 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548983 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548987 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548990 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548992 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 18:17:19.265005 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.548997 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549001 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549004 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549007 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549010 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549015 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549018 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549021 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549025 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549028 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549031 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549033 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549036 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549040 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549042 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549045 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549048 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549051 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549053 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549056 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 18:17:19.266167 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549059 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549061 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549064 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549067 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549070 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549073 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549076 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549080 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549084 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549088 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549091 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549094 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549097 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549100 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549104 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549107 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549109 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549114 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 18:17:19.267094 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:18.549117 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 18:17:19.268034 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.549122 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 18:17:19.268034 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.549238 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 12 18:17:19.268034 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.551595 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 12 18:17:19.268034 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.553134 2576 server.go:1019] "Starting client certificate rotation" Jul 12 18:17:19.268034 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.553240 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 12 18:17:19.268034 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.553299 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 12 18:17:19.268034 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.556651 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 12 18:17:19.268034 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.563229 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 12 18:17:19.268034 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.574822 2576 log.go:25] "Validated CRI v1 runtime API" Jul 12 18:17:19.268034 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.576954 2576 log.go:25] "Validated CRI v1 image API" Jul 12 18:17:19.268034 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.577810 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 12 18:17:19.268034 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.582877 2576 fs.go:135] Filesystem UUIDs: map[2a46bc7f-718e-44ca-9ca5-5fdab6776e23:/dev/nvme0n1p4 4c82b192-a549-4d97-852e-bad1c7e39efd:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 12 18:17:19.268734 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.582909 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 12 18:17:19.268734 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.584833 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.588325 2576 manager.go:216] Machine: {Timestamp:2026-07-12 18:17:18.58673818 +0000 UTC m=+0.354035755 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097631 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e91f7a04295e92b7c6eb44136eef3 SystemUUID:ec2e91f7-a042-95e9-2b7c-6eb44136eef3 BootID:cce2c206-ad14-48f3-ba94-98df2af51065 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ac:be:15:e7:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ac:be:15:e7:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:1b:3f:e1:14:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.588430 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.588511 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.592377 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.592405 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-5.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.593808 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.593820 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.593832 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.594328 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.595603 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.599702 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.599823 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.599857 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.599884 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.599900 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.599911 2576 kubelet.go:398] "Adding apiserver pod source" Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.599924 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 12 18:17:19.269402 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.600426 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.600446 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.602957 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.604538 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.604574 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.605291 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.605306 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.605312 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.605318 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.605326 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.605332 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.605485 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.605491 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.605501 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.605507 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.605516 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.605524 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.607207 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.607216 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.609330 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bqv5z" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.611280 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:18.611401 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:18.611466 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.612132 2576 server.go:1268] "Started kubelet" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.612208 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.612222 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.612257 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.613382 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.614640 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.617538 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bqv5z" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:18.618378 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-5.ec2.internal.18c19da028504520 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-5.ec2.internal,UID:ip-10-0-132-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-5.ec2.internal,},FirstTimestamp:2026-07-12 18:17:18.61210448 +0000 UTC m=+0.379402056,LastTimestamp:2026-07-12 18:17:18.61210448 +0000 UTC m=+0.379402056,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-5.ec2.internal,}" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:18.619671 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.620416 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.620510 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:18.622700 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-5.ec2.internal\" not found" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.623212 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.623225 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.623316 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.623399 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.623406 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 12 18:17:19.270339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.623777 2576 factory.go:55] Registering systemd factory Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.623793 2576 factory.go:223] Registration of the systemd container factory successfully Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.624021 2576 factory.go:153] Registering CRI-O factory Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.624041 2576 factory.go:223] Registration of the crio container factory successfully Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.624088 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.624108 2576 factory.go:103] Registering Raw factory Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.624118 2576 manager.go:1196] Started watching for new ooms in manager Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.624619 2576 manager.go:349] Starting recovery of all containers Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.634168 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.635462 2576 manager.go:354] Recovery completed Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:18.636803 2576 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-5.ec2.internal\" not found" node="ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.641448 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.643161 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasSufficientMemory" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.643188 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.643202 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasSufficientPID" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.643815 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.643826 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.643846 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.644978 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.645696 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasSufficientMemory" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.645721 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.645731 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasSufficientPID" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.646069 2576 policy_none.go:49] "None policy: Start" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.646083 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.646095 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.648169 2576 policy_none.go:47] "Start" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.689131 2576 manager.go:337] "Starting Device Plugin manager" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:18.689342 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.689358 2576 server.go:87] "Starting device plugin registration server" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.689568 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.689579 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.689679 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.689752 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.689759 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:18.690817 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:18.690884 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-5.ec2.internal\" not found" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.790567 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.791556 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasSufficientMemory" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.791586 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.791596 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasSufficientPID" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.791627 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.800120 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:18.800142 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-5.ec2.internal\": node \"ip-10-0-132-5.ec2.internal\" not found" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.809785 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.811194 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.811228 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.811250 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:18.811280 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.813960 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 18:17:19.272192 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:18.814865 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-5.ec2.internal\" not found" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.913468 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-5.ec2.internal"] Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.913570 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:18.915046 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-5.ec2.internal\" not found" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.915361 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasSufficientMemory" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.915392 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.915407 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasSufficientPID" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.918230 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.918366 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.918397 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.919043 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasSufficientMemory" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.919065 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.919076 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasSufficientPID" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.919085 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasSufficientMemory" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.919107 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.919116 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasSufficientPID" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.921361 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.921383 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.922084 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasSufficientMemory" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.922108 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.922122 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeHasSufficientPID" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.925714 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/64d9ab368b93d03e237759c2d650eb37-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal\" (UID: \"64d9ab368b93d03e237759c2d650eb37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.925751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64d9ab368b93d03e237759c2d650eb37-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal\" (UID: \"64d9ab368b93d03e237759c2d650eb37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:18.925811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4642eca106af85a646d7b3db5a92c5a2-config\") pod \"kube-apiserver-proxy-ip-10-0-132-5.ec2.internal\" (UID: \"4642eca106af85a646d7b3db5a92c5a2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:18.948545 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-5.ec2.internal\" not found" node="ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:18.953289 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-5.ec2.internal\" not found" node="ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:19.015357 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-5.ec2.internal\" not found" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.026845 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/64d9ab368b93d03e237759c2d650eb37-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal\" (UID: \"64d9ab368b93d03e237759c2d650eb37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.026872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64d9ab368b93d03e237759c2d650eb37-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal\" (UID: \"64d9ab368b93d03e237759c2d650eb37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.276220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.026890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4642eca106af85a646d7b3db5a92c5a2-config\") pod \"kube-apiserver-proxy-ip-10-0-132-5.ec2.internal\" (UID: \"4642eca106af85a646d7b3db5a92c5a2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.277184 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.026938 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4642eca106af85a646d7b3db5a92c5a2-config\") pod \"kube-apiserver-proxy-ip-10-0-132-5.ec2.internal\" (UID: \"4642eca106af85a646d7b3db5a92c5a2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.277184 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.026943 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/64d9ab368b93d03e237759c2d650eb37-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal\" (UID: \"64d9ab368b93d03e237759c2d650eb37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.277184 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.026977 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64d9ab368b93d03e237759c2d650eb37-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal\" (UID: \"64d9ab368b93d03e237759c2d650eb37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.277184 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:19.115801 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-5.ec2.internal\" not found" Jul 12 18:17:19.277184 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:19.217099 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-5.ec2.internal\" not found" Jul 12 18:17:19.277184 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.255049 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.277184 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.276142 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.507268 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:19.317166 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-5.ec2.internal\" not found" Jul 12 18:17:19.507268 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:19.420301 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-5.ec2.internal\" not found" Jul 12 18:17:19.763177 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:19.521420 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-5.ec2.internal\" not found" Jul 12 18:17:19.763177 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.552361 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 12 18:17:19.763177 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.552505 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 12 18:17:19.763177 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.552506 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 12 18:17:19.763177 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.614137 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 18:17:19.763177 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.623819 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.763177 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.623833 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 12 18:17:19.763177 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.624245 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-11 18:12:18 +0000 UTC" deadline="2028-01-29 00:48:01.255986872 +0000 UTC" Jul 12 18:17:19.763177 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.624288 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13566h30m41.631701621s" Jul 12 18:17:19.763177 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.633301 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 12 18:17:19.763177 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.635143 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal" Jul 12 18:17:19.763177 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.635708 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 12 18:17:19.763177 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.645222 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 12 18:17:19.763177 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.659967 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-f944q" Jul 12 18:17:19.763177 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.665378 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f944q" Jul 12 18:17:19.763177 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.751150 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 18:17:20.024412 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:19.895678 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d9ab368b93d03e237759c2d650eb37.slice/crio-0d53a94c3cf9c58b9b19d4cbb49b21848dda6df27536fcfb4feb57ed10f5b554 WatchSource:0}: Error finding container 0d53a94c3cf9c58b9b19d4cbb49b21848dda6df27536fcfb4feb57ed10f5b554: Status 404 returned error can't find the container with id 0d53a94c3cf9c58b9b19d4cbb49b21848dda6df27536fcfb4feb57ed10f5b554 Jul 12 18:17:20.024412 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:19.905528 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 18:17:20.024412 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:19.906296 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4642eca106af85a646d7b3db5a92c5a2.slice/crio-14fdd4531b2603599ea46b125b646347d7ad96c839009d0b5ee90edcfd154b64 WatchSource:0}: Error finding container 14fdd4531b2603599ea46b125b646347d7ad96c839009d0b5ee90edcfd154b64: Status 404 returned error can't find the container with id 14fdd4531b2603599ea46b125b646347d7ad96c839009d0b5ee90edcfd154b64 Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.602353 2576 apiserver.go:52] "Watching apiserver" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.612950 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.613386 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-g7w97","kube-system/konnectivity-agent-5fzsx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc","openshift-dns/node-resolver-rwd88","openshift-multus/network-metrics-daemon-jw4j5","openshift-network-operator/iptables-alerter-ghxpl","openshift-ovn-kubernetes/ovnkube-node-pq4vs","kube-system/global-pull-secret-syncer-pjl7r","kube-system/kube-apiserver-proxy-ip-10-0-132-5.ec2.internal","openshift-cluster-node-tuning-operator/tuned-n89rg","openshift-image-registry/node-ca-lpqkq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal","openshift-multus/multus-additional-cni-plugins-psnvh","openshift-multus/multus-plz8d"] Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.616072 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:20.616153 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.618419 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5fzsx" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.620928 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kc5mg\"" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.621564 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.621807 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.622070 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.623328 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rwd88" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.624256 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.624609 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rcsxx\"" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.624970 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.625158 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.625886 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:20.625954 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.628468 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.628756 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ghxpl" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.629397 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.629543 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tjnrl\"" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.630905 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.631350 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7zq8v\"" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.631422 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.631643 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.631784 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.634014 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.634039 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.634297 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:20.675135 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:20.634360 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.635636 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mv9gc\"" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.638198 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.638583 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.639379 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.640321 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.640679 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.641060 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.641195 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nsx7\" (UniqueName: \"kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7\") pod \"network-check-target-g7w97\" (UID: \"62fe9c7a-2477-4d68-b119-2dcc30e1c944\") " pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.641239 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-socket-dir\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.641275 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-registration-dir\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.641303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-etc-selinux\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.641332 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lpqkq" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.641382 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e7716b6-9736-4301-a491-6d236a12ed7c-agent-certs\") pod \"konnectivity-agent-5fzsx\" (UID: \"4e7716b6-9736-4301-a491-6d236a12ed7c\") " pod="kube-system/konnectivity-agent-5fzsx" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.641434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-device-dir\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.641500 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vjg5\" (UniqueName: \"kubernetes.io/projected/43a06d1a-c4e1-434b-b918-4d8edea89867-kube-api-access-5vjg5\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.641534 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmqcw\" (UniqueName: \"kubernetes.io/projected/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-kube-api-access-kmqcw\") pod \"network-metrics-daemon-jw4j5\" (UID: \"6a42ac1b-6e6a-4a60-8d10-42c2765b38ec\") " pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.641569 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e7716b6-9736-4301-a491-6d236a12ed7c-konnectivity-ca\") pod \"konnectivity-agent-5fzsx\" (UID: \"4e7716b6-9736-4301-a491-6d236a12ed7c\") " pod="kube-system/konnectivity-agent-5fzsx" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.641610 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-sys-fs\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.641646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/051fb076-c1be-4f39-bf98-7a0034d0e12e-tmp-dir\") pod \"node-resolver-rwd88\" (UID: \"051fb076-c1be-4f39-bf98-7a0034d0e12e\") " pod="openshift-dns/node-resolver-rwd88" Jul 12 18:17:20.677014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.641695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.641747 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/051fb076-c1be-4f39-bf98-7a0034d0e12e-hosts-file\") pod \"node-resolver-rwd88\" (UID: \"051fb076-c1be-4f39-bf98-7a0034d0e12e\") " pod="openshift-dns/node-resolver-rwd88" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.641781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlxxm\" (UniqueName: \"kubernetes.io/projected/051fb076-c1be-4f39-bf98-7a0034d0e12e-kube-api-access-qlxxm\") pod \"node-resolver-rwd88\" (UID: \"051fb076-c1be-4f39-bf98-7a0034d0e12e\") " pod="openshift-dns/node-resolver-rwd88" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.641809 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs\") pod \"network-metrics-daemon-jw4j5\" (UID: \"6a42ac1b-6e6a-4a60-8d10-42c2765b38ec\") " pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.642011 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.642149 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6ckg2\"" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.643601 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.644070 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.644110 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.644305 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.645591 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n9rmc\"" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.646812 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.648092 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.648093 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l7frw\"" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.648134 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.648144 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.648192 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.650195 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.652853 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bk9j6\"" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.653110 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.668188 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 18:12:19 +0000 UTC" deadline="2027-12-06 09:27:43.721107233 +0000 UTC" Jul 12 18:17:20.678271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.668232 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12279h10m23.05287891s" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.730150 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.730207 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.741992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/051fb076-c1be-4f39-bf98-7a0034d0e12e-hosts-file\") pod \"node-resolver-rwd88\" (UID: \"051fb076-c1be-4f39-bf98-7a0034d0e12e\") " pod="openshift-dns/node-resolver-rwd88" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-sysctl-d\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-systemd\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-tuned\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742121 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc40b8cb-8b6d-4358-b1ed-298429a488cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742134 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/051fb076-c1be-4f39-bf98-7a0034d0e12e-hosts-file\") pod \"node-resolver-rwd88\" (UID: \"051fb076-c1be-4f39-bf98-7a0034d0e12e\") " pod="openshift-dns/node-resolver-rwd88" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742153 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8n8f\" (UniqueName: \"kubernetes.io/projected/04a55769-f7ca-4dec-b5e1-3e2bce5421f6-kube-api-access-v8n8f\") pod \"iptables-alerter-ghxpl\" (UID: \"04a55769-f7ca-4dec-b5e1-3e2bce5421f6\") " pod="openshift-network-operator/iptables-alerter-ghxpl" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742180 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-systemd-units\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/03a2383e-2c1a-4e4e-afbc-6fd5f678e702-serviceca\") pod \"node-ca-lpqkq\" (UID: \"03a2383e-2c1a-4e4e-afbc-6fd5f678e702\") " pod="openshift-image-registry/node-ca-lpqkq" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742252 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-system-cni-dir\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-hostroot\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc40b8cb-8b6d-4358-b1ed-298429a488cd-cnibin\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-run-ovn\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmqcw\" (UniqueName: \"kubernetes.io/projected/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-kube-api-access-kmqcw\") pod \"network-metrics-daemon-jw4j5\" (UID: \"6a42ac1b-6e6a-4a60-8d10-42c2765b38ec\") " pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-var-lib-cni-multus\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.754974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l28j\" (UniqueName: \"kubernetes.io/projected/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-kube-api-access-8l28j\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.758284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc40b8cb-8b6d-4358-b1ed-298429a488cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.758284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742509 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a8808fa-07dc-4568-b10f-0a84be341957-kubelet-config\") pod \"global-pull-secret-syncer-pjl7r\" (UID: \"8a8808fa-07dc-4568-b10f-0a84be341957\") " pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:20.758284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-run-k8s-cni-cncf-io\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.758284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-run-netns\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.758284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-etc-kubernetes\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.758284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742612 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c63cd484-5494-4519-9bf0-35a9d9c84ceb-ovnkube-config\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.758284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742639 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c63cd484-5494-4519-9bf0-35a9d9c84ceb-ovn-node-metrics-cert\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.758284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742664 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c63cd484-5494-4519-9bf0-35a9d9c84ceb-ovnkube-script-lib\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.758284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742686 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-os-release\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.758284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742709 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-multus-conf-dir\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.758284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742734 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-modprobe-d\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.758284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-kubernetes\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.758284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-host\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.758284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-registration-dir\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.758284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742877 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-multus-cni-dir\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.758284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742915 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r629\" (UniqueName: \"kubernetes.io/projected/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-kube-api-access-8r629\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.759406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742951 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc40b8cb-8b6d-4358-b1ed-298429a488cd-system-cni-dir\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.759406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.742977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-cni-netd\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.759406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743003 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a8808fa-07dc-4568-b10f-0a84be341957-dbus\") pod \"global-pull-secret-syncer-pjl7r\" (UID: \"8a8808fa-07dc-4568-b10f-0a84be341957\") " pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:20.759406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-device-dir\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.759406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vjg5\" (UniqueName: \"kubernetes.io/projected/43a06d1a-c4e1-434b-b918-4d8edea89867-kube-api-access-5vjg5\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.759406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743132 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-registration-dir\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.759406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc40b8cb-8b6d-4358-b1ed-298429a488cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.759406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743179 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/04a55769-f7ca-4dec-b5e1-3e2bce5421f6-iptables-alerter-script\") pod \"iptables-alerter-ghxpl\" (UID: \"04a55769-f7ca-4dec-b5e1-3e2bce5421f6\") " pod="openshift-network-operator/iptables-alerter-ghxpl" Jul 12 18:17:20.759406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743182 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-device-dir\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.759406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-slash\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.759406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-etc-openvswitch\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.759406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e7716b6-9736-4301-a491-6d236a12ed7c-konnectivity-ca\") pod \"konnectivity-agent-5fzsx\" (UID: \"4e7716b6-9736-4301-a491-6d236a12ed7c\") " pod="kube-system/konnectivity-agent-5fzsx" Jul 12 18:17:20.759406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743334 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-var-lib-cni-bin\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.759406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743359 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-run-multus-certs\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.759406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743395 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f2cm\" (UniqueName: \"kubernetes.io/projected/fc40b8cb-8b6d-4358-b1ed-298429a488cd-kube-api-access-5f2cm\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.759406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-run-openvswitch\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.763375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743601 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c63cd484-5494-4519-9bf0-35a9d9c84ceb-env-overrides\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.763375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743657 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-multus-socket-dir-parent\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.763375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-var-lib-kubelet\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.763375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743736 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-multus-daemon-config\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.763375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743761 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-run-systemd\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.763375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-run-ovn-kubernetes\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.763375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nsx7\" (UniqueName: \"kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7\") pod \"network-check-target-g7w97\" (UID: \"62fe9c7a-2477-4d68-b119-2dcc30e1c944\") " pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:20.763375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743897 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-cni-binary-copy\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.763375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743922 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-tmp\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.763375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743953 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e7716b6-9736-4301-a491-6d236a12ed7c-agent-certs\") pod \"konnectivity-agent-5fzsx\" (UID: \"4e7716b6-9736-4301-a491-6d236a12ed7c\") " pod="kube-system/konnectivity-agent-5fzsx" Jul 12 18:17:20.763375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743984 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-var-lib-kubelet\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.763375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.743999 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e7716b6-9736-4301-a491-6d236a12ed7c-konnectivity-ca\") pod \"konnectivity-agent-5fzsx\" (UID: \"4e7716b6-9736-4301-a491-6d236a12ed7c\") " pod="kube-system/konnectivity-agent-5fzsx" Jul 12 18:17:20.763375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc40b8cb-8b6d-4358-b1ed-298429a488cd-os-release\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.763375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04a55769-f7ca-4dec-b5e1-3e2bce5421f6-host-slash\") pod \"iptables-alerter-ghxpl\" (UID: \"04a55769-f7ca-4dec-b5e1-3e2bce5421f6\") " pod="openshift-network-operator/iptables-alerter-ghxpl" Jul 12 18:17:20.763375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744135 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-kubelet\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.763375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/051fb076-c1be-4f39-bf98-7a0034d0e12e-tmp-dir\") pod \"node-resolver-rwd88\" (UID: \"051fb076-c1be-4f39-bf98-7a0034d0e12e\") " pod="openshift-dns/node-resolver-rwd88" Jul 12 18:17:20.763375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744187 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-cnibin\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.765008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-sys\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.765008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744238 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-var-lib-openvswitch\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.765008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-node-log\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.765008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.765008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.765008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxxm\" (UniqueName: \"kubernetes.io/projected/051fb076-c1be-4f39-bf98-7a0034d0e12e-kube-api-access-qlxxm\") pod \"node-resolver-rwd88\" (UID: \"051fb076-c1be-4f39-bf98-7a0034d0e12e\") " pod="openshift-dns/node-resolver-rwd88" Jul 12 18:17:20.765008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744384 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.765008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744398 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs\") pod \"network-metrics-daemon-jw4j5\" (UID: \"6a42ac1b-6e6a-4a60-8d10-42c2765b38ec\") " pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:20.765008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-sysctl-conf\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.765008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-socket-dir\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.765008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-etc-selinux\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.765008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744679 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-sysconfig\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.765008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744707 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-run\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.765008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc40b8cb-8b6d-4358-b1ed-298429a488cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.765008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-cni-bin\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.765008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744785 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret\") pod \"global-pull-secret-syncer-pjl7r\" (UID: \"8a8808fa-07dc-4568-b10f-0a84be341957\") " pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744820 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-lib-modules\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744841 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-etc-selinux\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-run-netns\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744930 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-log-socket\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-sys-fs\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.744985 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkjq8\" (UniqueName: \"kubernetes.io/projected/c63cd484-5494-4519-9bf0-35a9d9c84ceb-kube-api-access-hkjq8\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.745062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03a2383e-2c1a-4e4e-afbc-6fd5f678e702-host\") pod \"node-ca-lpqkq\" (UID: \"03a2383e-2c1a-4e4e-afbc-6fd5f678e702\") " pod="openshift-image-registry/node-ca-lpqkq" Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.745091 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k226\" (UniqueName: \"kubernetes.io/projected/03a2383e-2c1a-4e4e-afbc-6fd5f678e702-kube-api-access-9k226\") pod \"node-ca-lpqkq\" (UID: \"03a2383e-2c1a-4e4e-afbc-6fd5f678e702\") " pod="openshift-image-registry/node-ca-lpqkq" Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:20.745207 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.745252 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:20.745282 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs podName:6a42ac1b-6e6a-4a60-8d10-42c2765b38ec nodeName:}" failed. No retries permitted until 2026-07-12 18:17:21.245260435 +0000 UTC m=+3.012558018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs") pod "network-metrics-daemon-jw4j5" (UID: "6a42ac1b-6e6a-4a60-8d10-42c2765b38ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.745340 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/051fb076-c1be-4f39-bf98-7a0034d0e12e-tmp-dir\") pod \"node-resolver-rwd88\" (UID: \"051fb076-c1be-4f39-bf98-7a0034d0e12e\") " pod="openshift-dns/node-resolver-rwd88" Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.745475 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-sys-fs\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.745497 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43a06d1a-c4e1-434b-b918-4d8edea89867-socket-dir\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.750371 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e7716b6-9736-4301-a491-6d236a12ed7c-agent-certs\") pod \"konnectivity-agent-5fzsx\" (UID: \"4e7716b6-9736-4301-a491-6d236a12ed7c\") " pod="kube-system/konnectivity-agent-5fzsx" Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.757095 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmqcw\" (UniqueName: \"kubernetes.io/projected/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-kube-api-access-kmqcw\") pod \"network-metrics-daemon-jw4j5\" (UID: \"6a42ac1b-6e6a-4a60-8d10-42c2765b38ec\") " pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:20.757207 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 18:17:20.768405 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:20.757226 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 18:17:20.769653 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:20.757255 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2nsx7 for pod openshift-network-diagnostics/network-check-target-g7w97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 18:17:20.769653 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:20.757333 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7 podName:62fe9c7a-2477-4d68-b119-2dcc30e1c944 nodeName:}" failed. No retries permitted until 2026-07-12 18:17:21.257302962 +0000 UTC m=+3.024600538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2nsx7" (UniqueName: "kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7") pod "network-check-target-g7w97" (UID: "62fe9c7a-2477-4d68-b119-2dcc30e1c944") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 18:17:20.769653 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.758457 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vjg5\" (UniqueName: \"kubernetes.io/projected/43a06d1a-c4e1-434b-b918-4d8edea89867-kube-api-access-5vjg5\") pod \"aws-ebs-csi-driver-node-j52sc\" (UID: \"43a06d1a-c4e1-434b-b918-4d8edea89867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:20.769653 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.759110 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlxxm\" (UniqueName: \"kubernetes.io/projected/051fb076-c1be-4f39-bf98-7a0034d0e12e-kube-api-access-qlxxm\") pod \"node-resolver-rwd88\" (UID: \"051fb076-c1be-4f39-bf98-7a0034d0e12e\") " pod="openshift-dns/node-resolver-rwd88" Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.824148 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal" event={"ID":"64d9ab368b93d03e237759c2d650eb37","Type":"ContainerStarted","Data":"0d53a94c3cf9c58b9b19d4cbb49b21848dda6df27536fcfb4feb57ed10f5b554"} Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.825384 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-5.ec2.internal" event={"ID":"4642eca106af85a646d7b3db5a92c5a2","Type":"ContainerStarted","Data":"14fdd4531b2603599ea46b125b646347d7ad96c839009d0b5ee90edcfd154b64"} Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkjq8\" (UniqueName: \"kubernetes.io/projected/c63cd484-5494-4519-9bf0-35a9d9c84ceb-kube-api-access-hkjq8\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846100 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03a2383e-2c1a-4e4e-afbc-6fd5f678e702-host\") pod \"node-ca-lpqkq\" (UID: \"03a2383e-2c1a-4e4e-afbc-6fd5f678e702\") " pod="openshift-image-registry/node-ca-lpqkq" Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846125 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k226\" (UniqueName: \"kubernetes.io/projected/03a2383e-2c1a-4e4e-afbc-6fd5f678e702-kube-api-access-9k226\") pod \"node-ca-lpqkq\" (UID: \"03a2383e-2c1a-4e4e-afbc-6fd5f678e702\") " pod="openshift-image-registry/node-ca-lpqkq" Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846149 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-sysctl-d\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846175 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-systemd\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846190 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03a2383e-2c1a-4e4e-afbc-6fd5f678e702-host\") pod \"node-ca-lpqkq\" (UID: \"03a2383e-2c1a-4e4e-afbc-6fd5f678e702\") " pod="openshift-image-registry/node-ca-lpqkq" Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846198 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-tuned\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846223 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc40b8cb-8b6d-4358-b1ed-298429a488cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8n8f\" (UniqueName: \"kubernetes.io/projected/04a55769-f7ca-4dec-b5e1-3e2bce5421f6-kube-api-access-v8n8f\") pod \"iptables-alerter-ghxpl\" (UID: \"04a55769-f7ca-4dec-b5e1-3e2bce5421f6\") " pod="openshift-network-operator/iptables-alerter-ghxpl" Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-systemd-units\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/03a2383e-2c1a-4e4e-afbc-6fd5f678e702-serviceca\") pod \"node-ca-lpqkq\" (UID: \"03a2383e-2c1a-4e4e-afbc-6fd5f678e702\") " pod="openshift-image-registry/node-ca-lpqkq" Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846301 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-systemd\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-system-cni-dir\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846345 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-hostroot\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc40b8cb-8b6d-4358-b1ed-298429a488cd-cnibin\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.927747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-run-ovn\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846409 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-var-lib-cni-multus\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l28j\" (UniqueName: \"kubernetes.io/projected/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-kube-api-access-8l28j\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846442 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-sysctl-d\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846453 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-systemd-units\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846459 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc40b8cb-8b6d-4358-b1ed-298429a488cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846597 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-system-cni-dir\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846660 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-hostroot\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.846837 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc40b8cb-8b6d-4358-b1ed-298429a488cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847018 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/03a2383e-2c1a-4e4e-afbc-6fd5f678e702-serviceca\") pod \"node-ca-lpqkq\" (UID: \"03a2383e-2c1a-4e4e-afbc-6fd5f678e702\") " pod="openshift-image-registry/node-ca-lpqkq" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847113 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc40b8cb-8b6d-4358-b1ed-298429a488cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847144 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-run-ovn\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847149 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a8808fa-07dc-4568-b10f-0a84be341957-kubelet-config\") pod \"global-pull-secret-syncer-pjl7r\" (UID: \"8a8808fa-07dc-4568-b10f-0a84be341957\") " pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847179 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-run-k8s-cni-cncf-io\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-run-netns\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847259 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-run-k8s-cni-cncf-io\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-etc-kubernetes\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847303 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-etc-kubernetes\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.929733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847230 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-var-lib-cni-multus\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.930729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c63cd484-5494-4519-9bf0-35a9d9c84ceb-ovnkube-config\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.930729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847341 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc40b8cb-8b6d-4358-b1ed-298429a488cd-cnibin\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.930729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c63cd484-5494-4519-9bf0-35a9d9c84ceb-ovn-node-metrics-cert\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.930729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c63cd484-5494-4519-9bf0-35a9d9c84ceb-ovnkube-script-lib\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.930729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847395 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a8808fa-07dc-4568-b10f-0a84be341957-kubelet-config\") pod \"global-pull-secret-syncer-pjl7r\" (UID: \"8a8808fa-07dc-4568-b10f-0a84be341957\") " pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:20.930729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847409 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-os-release\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.930729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-multus-conf-dir\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.930729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847459 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-modprobe-d\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.930729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847492 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-kubernetes\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.930729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-host\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.930729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-multus-cni-dir\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.930729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r629\" (UniqueName: \"kubernetes.io/projected/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-kube-api-access-8r629\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.930729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847611 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc40b8cb-8b6d-4358-b1ed-298429a488cd-system-cni-dir\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.930729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-cni-netd\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.930729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a8808fa-07dc-4568-b10f-0a84be341957-dbus\") pod \"global-pull-secret-syncer-pjl7r\" (UID: \"8a8808fa-07dc-4568-b10f-0a84be341957\") " pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:20.930729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847696 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc40b8cb-8b6d-4358-b1ed-298429a488cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.930729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/04a55769-f7ca-4dec-b5e1-3e2bce5421f6-iptables-alerter-script\") pod \"iptables-alerter-ghxpl\" (UID: \"04a55769-f7ca-4dec-b5e1-3e2bce5421f6\") " pod="openshift-network-operator/iptables-alerter-ghxpl" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-slash\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-etc-openvswitch\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-var-lib-cni-bin\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-run-multus-certs\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847841 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c63cd484-5494-4519-9bf0-35a9d9c84ceb-ovnkube-config\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f2cm\" (UniqueName: \"kubernetes.io/projected/fc40b8cb-8b6d-4358-b1ed-298429a488cd-kube-api-access-5f2cm\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847914 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-run-netns\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-run-openvswitch\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c63cd484-5494-4519-9bf0-35a9d9c84ceb-env-overrides\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847978 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c63cd484-5494-4519-9bf0-35a9d9c84ceb-ovnkube-script-lib\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-multus-socket-dir-parent\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848035 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-multus-conf-dir\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-var-lib-kubelet\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848098 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-host\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848101 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-multus-socket-dir-parent\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848148 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-modprobe-d\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848198 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-kubernetes\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.931886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848221 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-var-lib-kubelet\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848234 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-multus-cni-dir\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-multus-daemon-config\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-run-systemd\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848302 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-slash\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848320 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-run-ovn-kubernetes\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848352 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-run-ovn-kubernetes\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848005 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-os-release\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-cni-binary-copy\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848404 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-tmp\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848397 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-run-systemd\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-var-lib-kubelet\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848461 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-run-multus-certs\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848464 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c63cd484-5494-4519-9bf0-35a9d9c84ceb-env-overrides\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.847955 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-run-openvswitch\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc40b8cb-8b6d-4358-b1ed-298429a488cd-os-release\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848503 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-etc-openvswitch\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04a55769-f7ca-4dec-b5e1-3e2bce5421f6-host-slash\") pod \"iptables-alerter-ghxpl\" (UID: \"04a55769-f7ca-4dec-b5e1-3e2bce5421f6\") " pod="openshift-network-operator/iptables-alerter-ghxpl" Jul 12 18:17:20.933331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-kubelet\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.934565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-cnibin\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.934565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-sys\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.934565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848618 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-var-lib-openvswitch\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.934565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-node-log\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.934565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.934565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-sysctl-conf\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.934565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848737 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-sysconfig\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.934565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848742 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-host-var-lib-cni-bin\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.934565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-run\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.934565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848762 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc40b8cb-8b6d-4358-b1ed-298429a488cd-os-release\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.934565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc40b8cb-8b6d-4358-b1ed-298429a488cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.934565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848813 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-cni-netd\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.934565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848859 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-cni-bin\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.934565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret\") pod \"global-pull-secret-syncer-pjl7r\" (UID: \"8a8808fa-07dc-4568-b10f-0a84be341957\") " pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:20.934565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848904 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/04a55769-f7ca-4dec-b5e1-3e2bce5421f6-iptables-alerter-script\") pod \"iptables-alerter-ghxpl\" (UID: \"04a55769-f7ca-4dec-b5e1-3e2bce5421f6\") " pod="openshift-network-operator/iptables-alerter-ghxpl" Jul 12 18:17:20.934565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-lib-modules\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.934565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848967 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-run-netns\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.935568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.848999 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-log-socket\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.935568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849004 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-lib-modules\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.935568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849079 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-var-lib-openvswitch\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.935568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849122 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-var-lib-kubelet\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.935568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849154 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04a55769-f7ca-4dec-b5e1-3e2bce5421f6-host-slash\") pod \"iptables-alerter-ghxpl\" (UID: \"04a55769-f7ca-4dec-b5e1-3e2bce5421f6\") " pod="openshift-network-operator/iptables-alerter-ghxpl" Jul 12 18:17:20.935568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849162 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a8808fa-07dc-4568-b10f-0a84be341957-dbus\") pod \"global-pull-secret-syncer-pjl7r\" (UID: \"8a8808fa-07dc-4568-b10f-0a84be341957\") " pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:20.935568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849185 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-kubelet\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.935568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849228 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-cnibin\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.935568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849250 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc40b8cb-8b6d-4358-b1ed-298429a488cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.935568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849270 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-sys\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.935568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849313 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-node-log\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.935568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849386 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-run-netns\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.935568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849455 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-cni-bin\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.935568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849497 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc40b8cb-8b6d-4358-b1ed-298429a488cd-system-cni-dir\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.935568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849545 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-log-socket\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.935568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849593 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c63cd484-5494-4519-9bf0-35a9d9c84ceb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.935568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849703 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc40b8cb-8b6d-4358-b1ed-298429a488cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:20.936964 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849727 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-sysctl-conf\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.936964 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849768 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-run\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.936964 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849794 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-multus-daemon-config\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.936964 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:20.849805 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 18:17:20.936964 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:20.849890 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret podName:8a8808fa-07dc-4568-b10f-0a84be341957 nodeName:}" failed. No retries permitted until 2026-07-12 18:17:21.349868813 +0000 UTC m=+3.117166395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret") pod "global-pull-secret-syncer-pjl7r" (UID: "8a8808fa-07dc-4568-b10f-0a84be341957") : object "kube-system"/"original-pull-secret" not registered Jul 12 18:17:20.936964 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.849814 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-sysconfig\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.936964 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.850079 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c63cd484-5494-4519-9bf0-35a9d9c84ceb-ovn-node-metrics-cert\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.936964 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.850109 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-etc-tuned\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.936964 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.850153 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-cni-binary-copy\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.936964 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.850999 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-tmp\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.936964 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.857485 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k226\" (UniqueName: \"kubernetes.io/projected/03a2383e-2c1a-4e4e-afbc-6fd5f678e702-kube-api-access-9k226\") pod \"node-ca-lpqkq\" (UID: \"03a2383e-2c1a-4e4e-afbc-6fd5f678e702\") " pod="openshift-image-registry/node-ca-lpqkq" Jul 12 18:17:20.936964 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.858464 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkjq8\" (UniqueName: \"kubernetes.io/projected/c63cd484-5494-4519-9bf0-35a9d9c84ceb-kube-api-access-hkjq8\") pod \"ovnkube-node-pq4vs\" (UID: \"c63cd484-5494-4519-9bf0-35a9d9c84ceb\") " pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:20.936964 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.859646 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l28j\" (UniqueName: \"kubernetes.io/projected/03b8837e-cff8-4cd8-8cf7-b32bda7762a7-kube-api-access-8l28j\") pod \"multus-plz8d\" (UID: \"03b8837e-cff8-4cd8-8cf7-b32bda7762a7\") " pod="openshift-multus/multus-plz8d" Jul 12 18:17:20.936964 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.859900 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r629\" (UniqueName: \"kubernetes.io/projected/00b3ef80-4e7a-4786-a483-a7d61b6d58b4-kube-api-access-8r629\") pod \"tuned-n89rg\" (UID: \"00b3ef80-4e7a-4786-a483-a7d61b6d58b4\") " pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:20.936964 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.860537 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8n8f\" (UniqueName: \"kubernetes.io/projected/04a55769-f7ca-4dec-b5e1-3e2bce5421f6-kube-api-access-v8n8f\") pod \"iptables-alerter-ghxpl\" (UID: \"04a55769-f7ca-4dec-b5e1-3e2bce5421f6\") " pod="openshift-network-operator/iptables-alerter-ghxpl" Jul 12 18:17:20.936964 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.861125 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f2cm\" (UniqueName: \"kubernetes.io/projected/fc40b8cb-8b6d-4358-b1ed-298429a488cd-kube-api-access-5f2cm\") pod \"multus-additional-cni-plugins-psnvh\" (UID: \"fc40b8cb-8b6d-4358-b1ed-298429a488cd\") " pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:21.087463 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.938660 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5fzsx" Jul 12 18:17:21.087463 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:20.951397 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7716b6_9736_4301_a491_6d236a12ed7c.slice/crio-4e0672e54773f66fbda13123674045f2d3373e3e34a46775ab2735cd2f0510c2 WatchSource:0}: Error finding container 4e0672e54773f66fbda13123674045f2d3373e3e34a46775ab2735cd2f0510c2: Status 404 returned error can't find the container with id 4e0672e54773f66fbda13123674045f2d3373e3e34a46775ab2735cd2f0510c2 Jul 12 18:17:21.087463 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.952798 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" Jul 12 18:17:21.087463 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.963705 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rwd88" Jul 12 18:17:21.087463 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:20.967766 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a06d1a_c4e1_434b_b918_4d8edea89867.slice/crio-c3a0c9b9ba7ca43805ff510d494c69c30d1d5cb578541c47ffd2d87b86c8e1ae WatchSource:0}: Error finding container c3a0c9b9ba7ca43805ff510d494c69c30d1d5cb578541c47ffd2d87b86c8e1ae: Status 404 returned error can't find the container with id c3a0c9b9ba7ca43805ff510d494c69c30d1d5cb578541c47ffd2d87b86c8e1ae Jul 12 18:17:21.087463 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:20.974511 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051fb076_c1be_4f39_bf98_7a0034d0e12e.slice/crio-b8feb05edd905f8b6b6d3c71e389129b3f5b103cffced84c6b96cd54abc9fc17 WatchSource:0}: Error finding container b8feb05edd905f8b6b6d3c71e389129b3f5b103cffced84c6b96cd54abc9fc17: Status 404 returned error can't find the container with id b8feb05edd905f8b6b6d3c71e389129b3f5b103cffced84c6b96cd54abc9fc17 Jul 12 18:17:21.087463 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.975876 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ghxpl" Jul 12 18:17:21.087463 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:20.987327 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a55769_f7ca_4dec_b5e1_3e2bce5421f6.slice/crio-d4fe42e3a4c94477327bc985ae97c5e81a8963602adbd5d2eece63794d551193 WatchSource:0}: Error finding container d4fe42e3a4c94477327bc985ae97c5e81a8963602adbd5d2eece63794d551193: Status 404 returned error can't find the container with id d4fe42e3a4c94477327bc985ae97c5e81a8963602adbd5d2eece63794d551193 Jul 12 18:17:21.087463 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:20.993066 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:21.087463 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:21.002788 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n89rg" Jul 12 18:17:21.087463 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:21.039094 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lpqkq" Jul 12 18:17:21.087463 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:21.041257 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-psnvh" Jul 12 18:17:21.087463 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:21.050156 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-plz8d" Jul 12 18:17:21.265196 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:21.094284 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 18:17:21.265196 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:21.251883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs\") pod \"network-metrics-daemon-jw4j5\" (UID: \"6a42ac1b-6e6a-4a60-8d10-42c2765b38ec\") " pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:21.265196 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:21.252045 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 18:17:21.265196 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:21.252129 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs podName:6a42ac1b-6e6a-4a60-8d10-42c2765b38ec nodeName:}" failed. No retries permitted until 2026-07-12 18:17:22.252109638 +0000 UTC m=+4.019407203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs") pod "network-metrics-daemon-jw4j5" (UID: "6a42ac1b-6e6a-4a60-8d10-42c2765b38ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 18:17:21.421123 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:21.352740 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nsx7\" (UniqueName: \"kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7\") pod \"network-check-target-g7w97\" (UID: \"62fe9c7a-2477-4d68-b119-2dcc30e1c944\") " pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:21.421123 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:21.352790 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret\") pod \"global-pull-secret-syncer-pjl7r\" (UID: \"8a8808fa-07dc-4568-b10f-0a84be341957\") " pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:21.421123 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:21.352924 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 18:17:21.421123 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:21.352938 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 18:17:21.421123 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:21.352957 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 18:17:21.421123 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:21.352971 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2nsx7 for pod openshift-network-diagnostics/network-check-target-g7w97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 18:17:21.421123 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:21.353004 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret podName:8a8808fa-07dc-4568-b10f-0a84be341957 nodeName:}" failed. No retries permitted until 2026-07-12 18:17:22.352985125 +0000 UTC m=+4.120282705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret") pod "global-pull-secret-syncer-pjl7r" (UID: "8a8808fa-07dc-4568-b10f-0a84be341957") : object "kube-system"/"original-pull-secret" not registered Jul 12 18:17:21.421123 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:21.353020 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7 podName:62fe9c7a-2477-4d68-b119-2dcc30e1c944 nodeName:}" failed. No retries permitted until 2026-07-12 18:17:22.353014065 +0000 UTC m=+4.120311627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2nsx7" (UniqueName: "kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7") pod "network-check-target-g7w97" (UID: "62fe9c7a-2477-4d68-b119-2dcc30e1c944") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 18:17:21.669433 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:21.669367 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 18:12:19 +0000 UTC" deadline="2027-12-31 02:12:45.388770215 +0000 UTC" Jul 12 18:17:21.669433 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:21.669429 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12871h55m23.719345829s" Jul 12 18:17:21.812789 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:21.812184 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:21.812789 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:21.812326 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:21.830188 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:21.830114 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rwd88" event={"ID":"051fb076-c1be-4f39-bf98-7a0034d0e12e","Type":"ContainerStarted","Data":"b8feb05edd905f8b6b6d3c71e389129b3f5b103cffced84c6b96cd54abc9fc17"} Jul 12 18:17:21.831945 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:21.831860 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" event={"ID":"43a06d1a-c4e1-434b-b918-4d8edea89867","Type":"ContainerStarted","Data":"c3a0c9b9ba7ca43805ff510d494c69c30d1d5cb578541c47ffd2d87b86c8e1ae"} Jul 12 18:17:21.833570 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:21.833544 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5fzsx" event={"ID":"4e7716b6-9736-4301-a491-6d236a12ed7c","Type":"ContainerStarted","Data":"4e0672e54773f66fbda13123674045f2d3373e3e34a46775ab2735cd2f0510c2"} Jul 12 18:17:21.835412 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:21.835360 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ghxpl" event={"ID":"04a55769-f7ca-4dec-b5e1-3e2bce5421f6","Type":"ContainerStarted","Data":"d4fe42e3a4c94477327bc985ae97c5e81a8963602adbd5d2eece63794d551193"} Jul 12 18:17:22.260170 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:22.260073 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs\") pod \"network-metrics-daemon-jw4j5\" (UID: \"6a42ac1b-6e6a-4a60-8d10-42c2765b38ec\") " pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:22.260351 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:22.260271 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 18:17:22.260351 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:22.260343 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs podName:6a42ac1b-6e6a-4a60-8d10-42c2765b38ec nodeName:}" failed. No retries permitted until 2026-07-12 18:17:24.260320408 +0000 UTC m=+6.027617990 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs") pod "network-metrics-daemon-jw4j5" (UID: "6a42ac1b-6e6a-4a60-8d10-42c2765b38ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 18:17:22.360754 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:22.360709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nsx7\" (UniqueName: \"kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7\") pod \"network-check-target-g7w97\" (UID: \"62fe9c7a-2477-4d68-b119-2dcc30e1c944\") " pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:22.360958 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:22.360799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret\") pod \"global-pull-secret-syncer-pjl7r\" (UID: \"8a8808fa-07dc-4568-b10f-0a84be341957\") " pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:22.361022 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:22.360982 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 18:17:22.361098 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:22.361049 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret podName:8a8808fa-07dc-4568-b10f-0a84be341957 nodeName:}" failed. No retries permitted until 2026-07-12 18:17:24.361031206 +0000 UTC m=+6.128328783 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret") pod "global-pull-secret-syncer-pjl7r" (UID: "8a8808fa-07dc-4568-b10f-0a84be341957") : object "kube-system"/"original-pull-secret" not registered Jul 12 18:17:22.361599 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:22.361479 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 18:17:22.361599 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:22.361504 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 18:17:22.361599 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:22.361517 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2nsx7 for pod openshift-network-diagnostics/network-check-target-g7w97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 18:17:22.361599 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:22.361563 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7 podName:62fe9c7a-2477-4d68-b119-2dcc30e1c944 nodeName:}" failed. No retries permitted until 2026-07-12 18:17:24.361548788 +0000 UTC m=+6.128846354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2nsx7" (UniqueName: "kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7") pod "network-check-target-g7w97" (UID: "62fe9c7a-2477-4d68-b119-2dcc30e1c944") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 18:17:22.580263 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:22.579052 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b3ef80_4e7a_4786_a483_a7d61b6d58b4.slice/crio-a01729fe98898876437f8a13322821627b2677420e4eed1fd6e4febf5c6739a9 WatchSource:0}: Error finding container a01729fe98898876437f8a13322821627b2677420e4eed1fd6e4febf5c6739a9: Status 404 returned error can't find the container with id a01729fe98898876437f8a13322821627b2677420e4eed1fd6e4febf5c6739a9 Jul 12 18:17:22.595386 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:22.594746 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc40b8cb_8b6d_4358_b1ed_298429a488cd.slice/crio-de611f9c4b7c71130021ef8995029ed1c74257f53d929b4460493c5e994000ee WatchSource:0}: Error finding container de611f9c4b7c71130021ef8995029ed1c74257f53d929b4460493c5e994000ee: Status 404 returned error can't find the container with id de611f9c4b7c71130021ef8995029ed1c74257f53d929b4460493c5e994000ee Jul 12 18:17:22.597706 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:17:22.597672 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b8837e_cff8_4cd8_8cf7_b32bda7762a7.slice/crio-fe0f9588cb4a9a9644996caf50801fdee7c783daf5ac01bf090854e7fd481c7c WatchSource:0}: Error finding container fe0f9588cb4a9a9644996caf50801fdee7c783daf5ac01bf090854e7fd481c7c: Status 404 returned error can't find the container with id fe0f9588cb4a9a9644996caf50801fdee7c783daf5ac01bf090854e7fd481c7c Jul 12 18:17:22.813087 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:22.812742 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:22.813496 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:22.812837 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:22.813496 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:22.813151 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:22.813496 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:22.813217 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:22.839587 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:22.839512 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lpqkq" event={"ID":"03a2383e-2c1a-4e4e-afbc-6fd5f678e702","Type":"ContainerStarted","Data":"5aabe853158882c8235b73a8460ebc57e5fe68c41435be6ad027257f4546f69d"} Jul 12 18:17:22.841618 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:22.841560 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n89rg" event={"ID":"00b3ef80-4e7a-4786-a483-a7d61b6d58b4","Type":"ContainerStarted","Data":"a01729fe98898876437f8a13322821627b2677420e4eed1fd6e4febf5c6739a9"} Jul 12 18:17:22.845109 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:22.845075 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psnvh" event={"ID":"fc40b8cb-8b6d-4358-b1ed-298429a488cd","Type":"ContainerStarted","Data":"de611f9c4b7c71130021ef8995029ed1c74257f53d929b4460493c5e994000ee"} Jul 12 18:17:22.848235 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:22.848204 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-5.ec2.internal" event={"ID":"4642eca106af85a646d7b3db5a92c5a2","Type":"ContainerStarted","Data":"322be30370ed7cc55e5e4fad492c7488000b79c7d69a4c1efa1ba58126b909ad"} Jul 12 18:17:22.849645 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:22.849577 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-plz8d" event={"ID":"03b8837e-cff8-4cd8-8cf7-b32bda7762a7","Type":"ContainerStarted","Data":"fe0f9588cb4a9a9644996caf50801fdee7c783daf5ac01bf090854e7fd481c7c"} Jul 12 18:17:22.852109 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:22.852029 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" event={"ID":"c63cd484-5494-4519-9bf0-35a9d9c84ceb","Type":"ContainerStarted","Data":"6f1998141ee74fafac3eab4205ec193b67ca654222453c12c7d244436067c5de"} Jul 12 18:17:22.863724 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:22.863663 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-5.ec2.internal" podStartSLOduration=3.863646301 podStartE2EDuration="3.863646301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 18:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:17:22.863175824 +0000 UTC m=+4.630473414" watchObservedRunningTime="2026-07-12 18:17:22.863646301 +0000 UTC m=+4.630943887" Jul 12 18:17:23.811984 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:23.811900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:23.812154 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:23.812043 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:23.858578 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:23.857292 2576 generic.go:356] "Generic (PLEG): container finished" podID="64d9ab368b93d03e237759c2d650eb37" containerID="8b7f4a3f49813e25e58a87e0fa92a0eb0e621094980f2d0f6eea8d876ee41faa" exitCode=0 Jul 12 18:17:23.858578 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:23.858325 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal" event={"ID":"64d9ab368b93d03e237759c2d650eb37","Type":"ContainerDied","Data":"8b7f4a3f49813e25e58a87e0fa92a0eb0e621094980f2d0f6eea8d876ee41faa"} Jul 12 18:17:24.276383 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:24.276293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs\") pod \"network-metrics-daemon-jw4j5\" (UID: \"6a42ac1b-6e6a-4a60-8d10-42c2765b38ec\") " pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:24.276559 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:24.276511 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 18:17:24.276624 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:24.276579 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs podName:6a42ac1b-6e6a-4a60-8d10-42c2765b38ec nodeName:}" failed. No retries permitted until 2026-07-12 18:17:28.276558572 +0000 UTC m=+10.043856135 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs") pod "network-metrics-daemon-jw4j5" (UID: "6a42ac1b-6e6a-4a60-8d10-42c2765b38ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 18:17:24.377063 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:24.376983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nsx7\" (UniqueName: \"kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7\") pod \"network-check-target-g7w97\" (UID: \"62fe9c7a-2477-4d68-b119-2dcc30e1c944\") " pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:24.377063 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:24.377064 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret\") pod \"global-pull-secret-syncer-pjl7r\" (UID: \"8a8808fa-07dc-4568-b10f-0a84be341957\") " pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:24.377331 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:24.377204 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 18:17:24.377331 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:24.377264 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret podName:8a8808fa-07dc-4568-b10f-0a84be341957 nodeName:}" failed. No retries permitted until 2026-07-12 18:17:28.377246349 +0000 UTC m=+10.144543914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret") pod "global-pull-secret-syncer-pjl7r" (UID: "8a8808fa-07dc-4568-b10f-0a84be341957") : object "kube-system"/"original-pull-secret" not registered Jul 12 18:17:24.377903 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:24.377724 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 18:17:24.377903 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:24.377801 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 18:17:24.377903 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:24.377817 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2nsx7 for pod openshift-network-diagnostics/network-check-target-g7w97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 18:17:24.377903 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:24.377877 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7 podName:62fe9c7a-2477-4d68-b119-2dcc30e1c944 nodeName:}" failed. No retries permitted until 2026-07-12 18:17:28.377860074 +0000 UTC m=+10.145157649 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2nsx7" (UniqueName: "kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7") pod "network-check-target-g7w97" (UID: "62fe9c7a-2477-4d68-b119-2dcc30e1c944") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 18:17:24.812469 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:24.811903 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:24.812469 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:24.812064 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:24.813011 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:24.812864 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:24.813011 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:24.812975 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:25.812245 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:25.811976 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:25.812245 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:25.812104 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:26.812978 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:26.812900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:26.813337 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:26.813036 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:26.814898 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:26.814874 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:26.815028 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:26.815007 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:27.812116 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:27.812036 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:27.812286 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:27.812171 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:28.313918 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:28.313883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs\") pod \"network-metrics-daemon-jw4j5\" (UID: \"6a42ac1b-6e6a-4a60-8d10-42c2765b38ec\") " pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:28.314378 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:28.314036 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 18:17:28.314378 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:28.314101 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs podName:6a42ac1b-6e6a-4a60-8d10-42c2765b38ec nodeName:}" failed. No retries permitted until 2026-07-12 18:17:36.314081105 +0000 UTC m=+18.081378666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs") pod "network-metrics-daemon-jw4j5" (UID: "6a42ac1b-6e6a-4a60-8d10-42c2765b38ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 18:17:28.415256 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:28.415212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nsx7\" (UniqueName: \"kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7\") pod \"network-check-target-g7w97\" (UID: \"62fe9c7a-2477-4d68-b119-2dcc30e1c944\") " pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:28.415427 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:28.415275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret\") pod \"global-pull-secret-syncer-pjl7r\" (UID: \"8a8808fa-07dc-4568-b10f-0a84be341957\") " pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:28.415427 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:28.415398 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 18:17:28.415533 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:28.415436 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 18:17:28.415533 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:28.415455 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret podName:8a8808fa-07dc-4568-b10f-0a84be341957 nodeName:}" failed. No retries permitted until 2026-07-12 18:17:36.415437674 +0000 UTC m=+18.182735251 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret") pod "global-pull-secret-syncer-pjl7r" (UID: "8a8808fa-07dc-4568-b10f-0a84be341957") : object "kube-system"/"original-pull-secret" not registered Jul 12 18:17:28.415533 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:28.415459 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 18:17:28.415533 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:28.415473 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2nsx7 for pod openshift-network-diagnostics/network-check-target-g7w97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 18:17:28.415533 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:28.415524 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7 podName:62fe9c7a-2477-4d68-b119-2dcc30e1c944 nodeName:}" failed. No retries permitted until 2026-07-12 18:17:36.415508592 +0000 UTC m=+18.182806153 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2nsx7" (UniqueName: "kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7") pod "network-check-target-g7w97" (UID: "62fe9c7a-2477-4d68-b119-2dcc30e1c944") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 18:17:28.815869 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:28.815357 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:28.815869 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:28.815473 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:28.816279 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:28.816123 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:28.816279 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:28.816241 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:29.812253 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:29.812218 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:29.812705 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:29.812357 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:30.814602 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:30.814004 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:30.814602 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:30.814144 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:30.814602 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:30.814517 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:30.815185 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:30.814618 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:31.812716 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:31.812417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:31.812931 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:31.812771 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:31.887489 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:31.887392 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rwd88" event={"ID":"051fb076-c1be-4f39-bf98-7a0034d0e12e","Type":"ContainerStarted","Data":"94deb550e344dd2536457a36d88ce1d21a2ec66e470bf7893a55de1ee546cfef"} Jul 12 18:17:31.889774 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:31.889745 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" event={"ID":"43a06d1a-c4e1-434b-b918-4d8edea89867","Type":"ContainerStarted","Data":"276895b8ff20de939b29497a321deb20a5363563b40aa5e0c1e97daa9b42fced"} Jul 12 18:17:31.892195 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:31.892169 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5fzsx" event={"ID":"4e7716b6-9736-4301-a491-6d236a12ed7c","Type":"ContainerStarted","Data":"65bf698e31ceea0f8106decc8c6d55c1f1d37eea94363dc022f14c8a92162d49"} Jul 12 18:17:31.893922 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:31.893868 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lpqkq" event={"ID":"03a2383e-2c1a-4e4e-afbc-6fd5f678e702","Type":"ContainerStarted","Data":"1074bfa68f8769be23ef176eb90983e7cf519a0f55df2d6ca188afc3ae8d4b85"} Jul 12 18:17:31.895865 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:31.895814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n89rg" event={"ID":"00b3ef80-4e7a-4786-a483-a7d61b6d58b4","Type":"ContainerStarted","Data":"ade7ef7060fe692c2f8cb21c0c5bfa69d111f02d148d27924624a8736447b3ab"} Jul 12 18:17:31.897963 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:31.897939 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc40b8cb-8b6d-4358-b1ed-298429a488cd" containerID="332412015ec32b719c8e779a63ce84622d7cbe8d18bb24b2ede0c6985a1c4be2" exitCode=0 Jul 12 18:17:31.898053 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:31.898030 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psnvh" event={"ID":"fc40b8cb-8b6d-4358-b1ed-298429a488cd","Type":"ContainerDied","Data":"332412015ec32b719c8e779a63ce84622d7cbe8d18bb24b2ede0c6985a1c4be2"} Jul 12 18:17:31.901372 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:31.901351 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal" event={"ID":"64d9ab368b93d03e237759c2d650eb37","Type":"ContainerStarted","Data":"6cb0d1133227fd04f498c31190de6edc30f0cc3307887dc84a5699649d0514b7"} Jul 12 18:17:31.920651 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:31.920598 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rwd88" podStartSLOduration=7.3821644840000005 podStartE2EDuration="12.9205798s" totalImagesPullingTime="5.538415316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 18:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:17:31.902440726 +0000 UTC m=+13.669738310" watchObservedRunningTime="2026-07-12 18:17:31.9205798 +0000 UTC m=+13.687877383" Jul 12 18:17:31.933723 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:31.933670 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n89rg" podStartSLOduration=4.465240473 podStartE2EDuration="12.933650932s" totalImagesPullingTime="8.468410459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 18:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:17:31.919800409 +0000 UTC m=+13.687097987" watchObservedRunningTime="2026-07-12 18:17:31.933650932 +0000 UTC m=+13.700948517" Jul 12 18:17:31.961890 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:31.961841 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lpqkq" podStartSLOduration=4.52381058 podStartE2EDuration="12.961821726s" totalImagesPullingTime="8.438011146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 18:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:17:31.933913471 +0000 UTC m=+13.701211057" watchObservedRunningTime="2026-07-12 18:17:31.961821726 +0000 UTC m=+13.729119312" Jul 12 18:17:31.978305 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:31.978257 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-5.ec2.internal" podStartSLOduration=12.978244788 podStartE2EDuration="12.978244788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 18:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:17:31.977992938 +0000 UTC m=+13.745290521" watchObservedRunningTime="2026-07-12 18:17:31.978244788 +0000 UTC m=+13.745542371" Jul 12 18:17:31.992028 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:31.991980 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5fzsx" podStartSLOduration=2.937414392 podStartE2EDuration="12.991966237s" totalImagesPullingTime="10.054551845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:17:31.991538877 +0000 UTC m=+13.758836461" watchObservedRunningTime="2026-07-12 18:17:31.991966237 +0000 UTC m=+13.759263819" Jul 12 18:17:32.153643 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:32.153554 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5fzsx" Jul 12 18:17:32.154303 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:32.154284 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5fzsx" Jul 12 18:17:32.811981 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:32.811905 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:32.812151 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:32.812044 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:32.812151 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:32.812083 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:32.812259 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:32.812158 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:32.905458 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:32.905401 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ghxpl" event={"ID":"04a55769-f7ca-4dec-b5e1-3e2bce5421f6","Type":"ContainerStarted","Data":"31479332e288be057e7867c7bc227f268352bbf8264fac3521ea32a98b2b1c9a"} Jul 12 18:17:32.919347 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:32.919294 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ghxpl" podStartSLOduration=7.447042618 podStartE2EDuration="13.919275998s" totalImagesPullingTime="6.47223338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 18:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:17:32.918754813 +0000 UTC m=+14.686052394" watchObservedRunningTime="2026-07-12 18:17:32.919275998 +0000 UTC m=+14.686573581" Jul 12 18:17:33.811475 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:33.811446 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:33.811638 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:33.811578 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:33.907705 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:33.907678 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 12 18:17:33.938702 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:33.938670 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5fzsx" Jul 12 18:17:33.939336 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:33.939309 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5fzsx" Jul 12 18:17:34.812299 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:34.812250 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:34.812483 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:34.812250 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:34.812483 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:34.812412 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:34.812579 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:34.812488 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:35.812381 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:35.812346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:35.813141 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:35.812478 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:36.383620 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:36.383583 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs\") pod \"network-metrics-daemon-jw4j5\" (UID: \"6a42ac1b-6e6a-4a60-8d10-42c2765b38ec\") " pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:36.383840 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:36.383723 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 18:17:36.383840 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:36.383817 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs podName:6a42ac1b-6e6a-4a60-8d10-42c2765b38ec nodeName:}" failed. No retries permitted until 2026-07-12 18:17:52.383779942 +0000 UTC m=+34.151077504 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs") pod "network-metrics-daemon-jw4j5" (UID: "6a42ac1b-6e6a-4a60-8d10-42c2765b38ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 18:17:36.484269 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:36.484005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret\") pod \"global-pull-secret-syncer-pjl7r\" (UID: \"8a8808fa-07dc-4568-b10f-0a84be341957\") " pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:36.484269 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:36.484100 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nsx7\" (UniqueName: \"kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7\") pod \"network-check-target-g7w97\" (UID: \"62fe9c7a-2477-4d68-b119-2dcc30e1c944\") " pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:36.484269 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:36.484161 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 18:17:36.484269 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:36.484237 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 18:17:36.484269 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:36.484267 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 18:17:36.484269 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:36.484279 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2nsx7 for pod openshift-network-diagnostics/network-check-target-g7w97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 18:17:36.484703 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:36.484241 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret podName:8a8808fa-07dc-4568-b10f-0a84be341957 nodeName:}" failed. No retries permitted until 2026-07-12 18:17:52.484218772 +0000 UTC m=+34.251516357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret") pod "global-pull-secret-syncer-pjl7r" (UID: "8a8808fa-07dc-4568-b10f-0a84be341957") : object "kube-system"/"original-pull-secret" not registered Jul 12 18:17:36.484703 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:36.484341 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7 podName:62fe9c7a-2477-4d68-b119-2dcc30e1c944 nodeName:}" failed. No retries permitted until 2026-07-12 18:17:52.48432673 +0000 UTC m=+34.251624292 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2nsx7" (UniqueName: "kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7") pod "network-check-target-g7w97" (UID: "62fe9c7a-2477-4d68-b119-2dcc30e1c944") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 18:17:36.812104 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:36.812025 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:36.812253 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:36.812170 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:36.812570 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:36.812532 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:36.812924 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:36.812633 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:37.410061 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:37.410022 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:37.497905 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:37.497868 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:37.557555 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:37.557494 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:37.696373 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:37.696263 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:37.796435 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:37.796407 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:37.811889 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:37.811854 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:37.812045 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:37.811989 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:38.070484 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:38.070407 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:38.672017 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:38.671985 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:38.720969 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:38.720936 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:38.812884 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:38.812855 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:38.813047 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:38.812956 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:38.813047 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:38.813036 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:38.813136 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:38.813114 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:39.811966 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:39.811932 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:39.812407 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:39.812059 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:40.811592 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:40.811554 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:40.811765 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:40.811685 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:40.811765 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:40.811726 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:40.811900 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:40.811851 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:41.716284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:41.716249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:41.770455 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:41.770237 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:41.812285 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:41.812249 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:41.812477 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:41.812380 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:42.620291 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:42.620264 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 12 18:17:42.705243 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:42.705146 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-12T18:17:42.62028572Z","UUID":"839398d4-fbfd-4abc-88ec-8fe5997a4fb4","Handler":null,"Name":"","Endpoint":""} Jul 12 18:17:42.706856 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:42.706837 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 12 18:17:42.706965 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:42.706876 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 12 18:17:42.811917 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:42.811540 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:42.811917 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:42.811538 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:42.811917 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:42.811685 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:42.811917 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:42.811727 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:42.928122 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:42.927958 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" event={"ID":"43a06d1a-c4e1-434b-b918-4d8edea89867","Type":"ContainerStarted","Data":"1f8f947e6125b5ae11489676a555b6ccd62d8d57af265f0ef9411110d96d49d0"} Jul 12 18:17:42.929375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:42.929355 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-plz8d" event={"ID":"03b8837e-cff8-4cd8-8cf7-b32bda7762a7","Type":"ContainerStarted","Data":"8037242965c24d722b5d6405b400a2e2e704bd6b509a2b5ec4ad690698e4f2ae"} Jul 12 18:17:42.931004 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:42.930982 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" event={"ID":"c63cd484-5494-4519-9bf0-35a9d9c84ceb","Type":"ContainerStarted","Data":"d806434fcc6a3790585ff2b5c2eed66157763b5fe53fc077910767df42349022"} Jul 12 18:17:42.931073 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:42.931058 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" event={"ID":"c63cd484-5494-4519-9bf0-35a9d9c84ceb","Type":"ContainerStarted","Data":"a8b87e4a400698827f1635dc896600472600811c8cfbf77283e64343fddc432e"} Jul 12 18:17:42.946970 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:42.946932 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-plz8d" podStartSLOduration=4.038796548 podStartE2EDuration="23.946917475s" totalImagesPullingTime="19.908120927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 18:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:17:42.946317842 +0000 UTC m=+24.713615424" watchObservedRunningTime="2026-07-12 18:17:42.946917475 +0000 UTC m=+24.714215057" Jul 12 18:17:43.065103 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:43.065073 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:43.165360 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:43.165333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:43.812243 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:43.812178 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:43.812727 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:43.812268 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:43.936174 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:43.936139 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" event={"ID":"c63cd484-5494-4519-9bf0-35a9d9c84ceb","Type":"ContainerStarted","Data":"07012fb1985249c7d65478526d8bbc9053dbe987fe738111ab8bd73cc732ee53"} Jul 12 18:17:43.936343 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:43.936219 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" event={"ID":"c63cd484-5494-4519-9bf0-35a9d9c84ceb","Type":"ContainerStarted","Data":"7e52edff55c4e001a549ac7a532b5d257758957a2e7c9359d6f18110987c839a"} Jul 12 18:17:43.936343 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:43.936263 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" event={"ID":"c63cd484-5494-4519-9bf0-35a9d9c84ceb","Type":"ContainerStarted","Data":"e6fdbffdf7b251deff2c1783f47466896ac10eb6675eda05bde2c0e2065a9ae6"} Jul 12 18:17:43.936452 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:43.936358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" event={"ID":"c63cd484-5494-4519-9bf0-35a9d9c84ceb","Type":"ContainerStarted","Data":"b2a707eeed38ba52d31b64d2c6fe0aaa6b1f3ddb7d801ebf1935eb5f49114f2a"} Jul 12 18:17:43.938070 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:43.938032 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" event={"ID":"43a06d1a-c4e1-434b-b918-4d8edea89867","Type":"ContainerStarted","Data":"382cd7945a5857fe7f4fd852ab5d40c76eddabf38383dd224bc423f33d874e3a"} Jul 12 18:17:43.939702 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:43.939677 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc40b8cb-8b6d-4358-b1ed-298429a488cd" containerID="dd5ddcdf25966896d4e97e2e5e9289a40f61b5ddb4b8dade0adcb4bbbc19fe50" exitCode=0 Jul 12 18:17:43.939816 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:43.939763 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psnvh" event={"ID":"fc40b8cb-8b6d-4358-b1ed-298429a488cd","Type":"ContainerDied","Data":"dd5ddcdf25966896d4e97e2e5e9289a40f61b5ddb4b8dade0adcb4bbbc19fe50"} Jul 12 18:17:43.955971 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:43.955938 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j52sc" podStartSLOduration=8.750043984 podStartE2EDuration="24.955926743s" totalImagesPullingTime="16.205882759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 18:17:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:17:43.954442622 +0000 UTC m=+25.721740203" watchObservedRunningTime="2026-07-12 18:17:43.955926743 +0000 UTC m=+25.723224325" Jul 12 18:17:44.811521 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:44.811492 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:44.811865 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:44.811610 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:44.811865 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:44.811654 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:44.811865 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:44.811751 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:45.812435 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:45.812403 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:45.812877 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:45.812502 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:45.946343 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:45.946307 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc40b8cb-8b6d-4358-b1ed-298429a488cd" containerID="4838efbb749dfaa633ae882ae4772a6dde3ff2bc291a327025ff6f4efbee662d" exitCode=0 Jul 12 18:17:45.946524 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:45.946381 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psnvh" event={"ID":"fc40b8cb-8b6d-4358-b1ed-298429a488cd","Type":"ContainerDied","Data":"4838efbb749dfaa633ae882ae4772a6dde3ff2bc291a327025ff6f4efbee662d"} Jul 12 18:17:45.949735 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:45.949715 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" event={"ID":"c63cd484-5494-4519-9bf0-35a9d9c84ceb","Type":"ContainerStarted","Data":"802837695a6478fca404e4a2f1da02643431c66b42119f2c9eb9023ecc24f884"} Jul 12 18:17:46.812276 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:46.812204 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:46.812432 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:46.812308 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:46.812432 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:46.812338 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:46.812432 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:46.812386 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:47.812550 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:47.812313 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:47.812982 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:47.812562 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:47.956586 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:47.956549 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc40b8cb-8b6d-4358-b1ed-298429a488cd" containerID="597533aa56ce70a272af4c1889bd5e988e8254c7901af2cc02999c2dd1059884" exitCode=0 Jul 12 18:17:47.956763 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:47.956646 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psnvh" event={"ID":"fc40b8cb-8b6d-4358-b1ed-298429a488cd","Type":"ContainerDied","Data":"597533aa56ce70a272af4c1889bd5e988e8254c7901af2cc02999c2dd1059884"} Jul 12 18:17:47.960549 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:47.960523 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" event={"ID":"c63cd484-5494-4519-9bf0-35a9d9c84ceb","Type":"ContainerStarted","Data":"6aa36f627089aa6a782b1f1d676ac88161bd0ec574151ebba6f1ca8f1c26aa15"} Jul 12 18:17:47.960834 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:47.960814 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:47.960918 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:47.960891 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:47.960966 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:47.960940 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:47.976092 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:47.976071 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:47.976180 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:47.976164 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:17:47.999902 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:47.999859 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" podStartSLOduration=9.08577091 podStartE2EDuration="28.999844499s" totalImagesPullingTime="19.914073589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 18:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:17:47.999073176 +0000 UTC m=+29.766370767" watchObservedRunningTime="2026-07-12 18:17:47.999844499 +0000 UTC m=+29.767142080" Jul 12 18:17:48.541950 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:48.541915 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:48.592531 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:48.592485 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:48.813277 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:48.813202 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:48.813726 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:48.813322 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:48.813726 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:48.813379 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:48.813726 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:48.813512 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:49.454427 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:49.454392 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:49.554388 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:49.554358 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:49.763363 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:49.763279 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g7w97"] Jul 12 18:17:49.763531 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:49.763404 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:49.763531 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:49.763506 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:49.764021 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:49.763995 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jw4j5"] Jul 12 18:17:49.764156 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:49.764085 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:49.764220 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:49.764159 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:49.764967 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:49.764930 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pjl7r"] Jul 12 18:17:49.765065 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:49.765001 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:49.765110 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:49.765073 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:51.812137 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:51.812061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:51.812569 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:51.812176 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:51.812569 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:51.812187 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:51.812569 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:51.812212 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:51.812569 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:51.812321 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:51.812569 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:51.812467 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:51.866457 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:51.866422 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:51.930649 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:51.930618 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:51.991383 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:51.991343 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:52.403877 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:52.403671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs\") pod \"network-metrics-daemon-jw4j5\" (UID: \"6a42ac1b-6e6a-4a60-8d10-42c2765b38ec\") " pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:52.403877 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:52.403820 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 18:17:52.404091 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:52.403943 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs podName:6a42ac1b-6e6a-4a60-8d10-42c2765b38ec nodeName:}" failed. No retries permitted until 2026-07-12 18:18:24.403924002 +0000 UTC m=+66.171221575 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs") pod "network-metrics-daemon-jw4j5" (UID: "6a42ac1b-6e6a-4a60-8d10-42c2765b38ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 18:17:52.504722 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:52.504695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret\") pod \"global-pull-secret-syncer-pjl7r\" (UID: \"8a8808fa-07dc-4568-b10f-0a84be341957\") " pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:52.504880 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:52.504746 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nsx7\" (UniqueName: \"kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7\") pod \"network-check-target-g7w97\" (UID: \"62fe9c7a-2477-4d68-b119-2dcc30e1c944\") " pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:52.504880 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:52.504867 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 18:17:52.504967 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:52.504892 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 18:17:52.504967 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:52.504910 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 18:17:52.504967 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:52.504924 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2nsx7 for pod openshift-network-diagnostics/network-check-target-g7w97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 18:17:52.504967 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:52.504946 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret podName:8a8808fa-07dc-4568-b10f-0a84be341957 nodeName:}" failed. No retries permitted until 2026-07-12 18:18:24.504916133 +0000 UTC m=+66.272213694 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret") pod "global-pull-secret-syncer-pjl7r" (UID: "8a8808fa-07dc-4568-b10f-0a84be341957") : object "kube-system"/"original-pull-secret" not registered Jul 12 18:17:52.504967 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:52.504967 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7 podName:62fe9c7a-2477-4d68-b119-2dcc30e1c944 nodeName:}" failed. No retries permitted until 2026-07-12 18:18:24.504954657 +0000 UTC m=+66.272252223 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2nsx7" (UniqueName: "kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7") pod "network-check-target-g7w97" (UID: "62fe9c7a-2477-4d68-b119-2dcc30e1c944") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 18:17:52.975304 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:52.975274 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc40b8cb-8b6d-4358-b1ed-298429a488cd" containerID="da6cf0559ab55f3e73f1a974544aa220cc226b9559617f8d7006b8a4dadb92a3" exitCode=0 Jul 12 18:17:52.975811 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:52.975325 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psnvh" event={"ID":"fc40b8cb-8b6d-4358-b1ed-298429a488cd","Type":"ContainerDied","Data":"da6cf0559ab55f3e73f1a974544aa220cc226b9559617f8d7006b8a4dadb92a3"} Jul 12 18:17:53.153818 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:53.153768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:53.207366 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:53.207333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:53.578498 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:53.578467 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:53.629683 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:53.629652 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:53.811804 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:53.811756 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:53.811936 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:53.811815 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:53.811936 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:53.811766 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:53.811936 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:53.811892 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:53.812062 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:53.811953 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:53.812062 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:53.812044 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:53.980673 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:53.980588 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc40b8cb-8b6d-4358-b1ed-298429a488cd" containerID="11d85538e0c47ebba80dfa8b1912579eb6687bddde4974f61a86a0e8dca2df41" exitCode=0 Jul 12 18:17:53.981148 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:53.980665 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psnvh" event={"ID":"fc40b8cb-8b6d-4358-b1ed-298429a488cd","Type":"ContainerDied","Data":"11d85538e0c47ebba80dfa8b1912579eb6687bddde4974f61a86a0e8dca2df41"} Jul 12 18:17:54.011245 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:54.011212 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:54.111678 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:54.111646 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:54.287450 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:54.287368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:54.388181 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:54.388145 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:54.500274 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:54.500245 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:54.597681 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:54.597656 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:54.986633 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:54.986539 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psnvh" event={"ID":"fc40b8cb-8b6d-4358-b1ed-298429a488cd","Type":"ContainerStarted","Data":"87655ce21df41301ab20ca24629375445cc9bbfdf4118251b066adef16cc2665"} Jul 12 18:17:55.008611 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:55.008565 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-psnvh" podStartSLOduration=10.41406007 podStartE2EDuration="36.00855139s" totalImagesPullingTime="25.59449132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 18:17:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:17:55.007992189 +0000 UTC m=+36.775289771" watchObservedRunningTime="2026-07-12 18:17:55.00855139 +0000 UTC m=+36.775848972" Jul 12 18:17:55.125886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:55.125854 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:55.178624 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:55.178598 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:55.517922 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:55.517891 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rwd88_051fb076-c1be-4f39-bf98-7a0034d0e12e/dns-node-resolver/0.log" Jul 12 18:17:55.812503 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:55.812420 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:55.812738 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:55.812546 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:55.812738 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:55.812561 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjl7r" podUID="8a8808fa-07dc-4568-b10f-0a84be341957" Jul 12 18:17:55.812738 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:55.812578 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:55.812738 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:55.812668 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7w97" podUID="62fe9c7a-2477-4d68-b119-2dcc30e1c944" Jul 12 18:17:55.812969 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:17:55.812748 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw4j5" podUID="6a42ac1b-6e6a-4a60-8d10-42c2765b38ec" Jul 12 18:17:55.918021 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:55.917993 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:56.454460 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:56.454428 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-5.ec2.internal" event="NodeReady" Jul 12 18:17:56.454843 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:56.454561 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 12 18:17:56.517957 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:56.517929 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lpqkq_03a2383e-2c1a-4e4e-afbc-6fd5f678e702/node-ca/0.log" Jul 12 18:17:56.765517 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:56.765441 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:17:57.812234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:57.812194 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:17:57.812234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:57.812230 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:17:57.812703 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:57.812293 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:17:57.816824 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:57.816781 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 12 18:17:57.816962 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:57.816880 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 12 18:17:57.816962 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:57.816820 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p9cbp\"" Jul 12 18:17:57.816962 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:57.816926 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 12 18:17:57.816962 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:57.816827 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hb5mg\"" Jul 12 18:17:57.817184 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:57.816817 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 12 18:17:59.886776 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:17:59.886742 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:00.113219 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:00.113185 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:00.214472 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:00.214392 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:00.356878 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:00.356840 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:00.457689 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:00.457641 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:00.557002 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:00.556919 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:01.004943 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:01.004906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:01.607678 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:01.607641 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:02.207597 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:02.207559 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:02.407801 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:02.407768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:02.607527 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:02.607492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:03.008452 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:03.008374 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:03.201492 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:03.201455 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:03.734574 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:03.734537 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:03.835238 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:03.835208 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:04.008008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:04.007931 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:04.108924 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:04.108888 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:04.422546 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:04.422516 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:04.610361 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:04.610332 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:04.665445 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:04.665407 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:05.202730 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:05.202692 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:05.303115 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:05.303068 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:05.609994 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:05.609964 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:05.710150 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:05.710125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:05.759658 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:05.759633 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:06.008957 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:06.008878 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:06.406511 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:06.406483 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:06.514577 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:06.514547 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:06.618365 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:06.618334 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:07.212144 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:07.212116 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:07.339707 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:07.339671 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:07.444479 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:07.444441 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:07.611539 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:07.611515 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:08.611429 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:08.611394 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:09.006703 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:09.006613 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:09.208389 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:09.208354 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:09.413802 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:09.413756 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:09.606613 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:09.606570 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:10.407985 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:10.407950 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:10.605860 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:10.605829 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:10.807606 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:10.807528 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:11.559963 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:11.559928 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:11.722857 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:11.722827 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:11.802259 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:11.802229 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:15.651154 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:15.651112 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:15.752215 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:15.752182 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:18.149098 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.149055 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wndcz"] Jul 12 18:18:18.151500 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.151483 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-c2zhg"] Jul 12 18:18:18.151622 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.151605 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wndcz" Jul 12 18:18:18.152887 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.152867 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c2zhg" Jul 12 18:18:18.154335 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.154315 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 12 18:18:18.154616 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.154601 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h7xmt\"" Jul 12 18:18:18.154674 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.154659 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 12 18:18:18.155307 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.155292 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 12 18:18:18.155433 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.155416 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 12 18:18:18.155493 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.155430 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 12 18:18:18.155727 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.155712 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hwzkn\"" Jul 12 18:18:18.162320 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.162291 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c2zhg"] Jul 12 18:18:18.166262 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.166241 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wndcz"] Jul 12 18:18:18.180620 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.180590 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4df2da2e-9018-4a69-8776-f152cd0b6052-metrics-tls\") pod \"dns-default-wndcz\" (UID: \"4df2da2e-9018-4a69-8776-f152cd0b6052\") " pod="openshift-dns/dns-default-wndcz" Jul 12 18:18:18.180620 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.180623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70b7d9dd-771d-4d6c-be24-149e3fa53de0-cert\") pod \"ingress-canary-c2zhg\" (UID: \"70b7d9dd-771d-4d6c-be24-149e3fa53de0\") " pod="openshift-ingress-canary/ingress-canary-c2zhg" Jul 12 18:18:18.180831 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.180642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4df2da2e-9018-4a69-8776-f152cd0b6052-tmp-dir\") pod \"dns-default-wndcz\" (UID: \"4df2da2e-9018-4a69-8776-f152cd0b6052\") " pod="openshift-dns/dns-default-wndcz" Jul 12 18:18:18.180831 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.180756 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4df2da2e-9018-4a69-8776-f152cd0b6052-config-volume\") pod \"dns-default-wndcz\" (UID: \"4df2da2e-9018-4a69-8776-f152cd0b6052\") " pod="openshift-dns/dns-default-wndcz" Jul 12 18:18:18.180831 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.180806 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59nps\" (UniqueName: \"kubernetes.io/projected/70b7d9dd-771d-4d6c-be24-149e3fa53de0-kube-api-access-59nps\") pod \"ingress-canary-c2zhg\" (UID: \"70b7d9dd-771d-4d6c-be24-149e3fa53de0\") " pod="openshift-ingress-canary/ingress-canary-c2zhg" Jul 12 18:18:18.180831 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.180824 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l5mw\" (UniqueName: \"kubernetes.io/projected/4df2da2e-9018-4a69-8776-f152cd0b6052-kube-api-access-5l5mw\") pod \"dns-default-wndcz\" (UID: \"4df2da2e-9018-4a69-8776-f152cd0b6052\") " pod="openshift-dns/dns-default-wndcz" Jul 12 18:18:18.182236 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.182216 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-88mc5"] Jul 12 18:18:18.184188 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.184174 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-88mc5" Jul 12 18:18:18.187094 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.187068 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9cwfc\"" Jul 12 18:18:18.187212 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.187116 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 12 18:18:18.187212 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.187176 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 12 18:18:18.187212 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.187191 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 12 18:18:18.187212 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.187196 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 12 18:18:18.195094 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.195072 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-88mc5"] Jul 12 18:18:18.195242 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.195223 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:18.249760 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.249726 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vths9"] Jul 12 18:18:18.251641 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.251625 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f788c444b-9sjcx"] Jul 12 18:18:18.251794 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.251765 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vths9" Jul 12 18:18:18.253412 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.253390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.254327 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.254305 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hm8zq\"" Jul 12 18:18:18.254327 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.254321 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 12 18:18:18.255729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.255715 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p9vq2\"" Jul 12 18:18:18.255842 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.255728 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 12 18:18:18.255842 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.255740 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 12 18:18:18.255993 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.255980 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 12 18:18:18.261440 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.261420 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 12 18:18:18.264705 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.264687 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vths9"] Jul 12 18:18:18.265438 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.265422 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f788c444b-9sjcx"] Jul 12 18:18:18.281206 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281181 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b554cd4b-708d-47ab-b6c2-76ae2587ee31-trusted-ca\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.281316 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4df2da2e-9018-4a69-8776-f152cd0b6052-config-volume\") pod \"dns-default-wndcz\" (UID: \"4df2da2e-9018-4a69-8776-f152cd0b6052\") " pod="openshift-dns/dns-default-wndcz" Jul 12 18:18:18.281316 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281231 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88mc5\" (UID: \"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2\") " pod="openshift-insights/insights-runtime-extractor-88mc5" Jul 12 18:18:18.281316 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281248 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/16e8851a-ed17-47d0-b6d4-f998ea699fc7-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-vths9\" (UID: \"16e8851a-ed17-47d0-b6d4-f998ea699fc7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vths9" Jul 12 18:18:18.281316 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281305 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b554cd4b-708d-47ab-b6c2-76ae2587ee31-bound-sa-token\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.281456 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281396 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59nps\" (UniqueName: \"kubernetes.io/projected/70b7d9dd-771d-4d6c-be24-149e3fa53de0-kube-api-access-59nps\") pod \"ingress-canary-c2zhg\" (UID: \"70b7d9dd-771d-4d6c-be24-149e3fa53de0\") " pod="openshift-ingress-canary/ingress-canary-c2zhg" Jul 12 18:18:18.281456 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281427 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l5mw\" (UniqueName: \"kubernetes.io/projected/4df2da2e-9018-4a69-8776-f152cd0b6052-kube-api-access-5l5mw\") pod \"dns-default-wndcz\" (UID: \"4df2da2e-9018-4a69-8776-f152cd0b6052\") " pod="openshift-dns/dns-default-wndcz" Jul 12 18:18:18.281561 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4df2da2e-9018-4a69-8776-f152cd0b6052-metrics-tls\") pod \"dns-default-wndcz\" (UID: \"4df2da2e-9018-4a69-8776-f152cd0b6052\") " pod="openshift-dns/dns-default-wndcz" Jul 12 18:18:18.281608 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b554cd4b-708d-47ab-b6c2-76ae2587ee31-image-registry-private-configuration\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.281650 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281614 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b554cd4b-708d-47ab-b6c2-76ae2587ee31-registry-certificates\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.281688 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70b7d9dd-771d-4d6c-be24-149e3fa53de0-cert\") pod \"ingress-canary-c2zhg\" (UID: \"70b7d9dd-771d-4d6c-be24-149e3fa53de0\") " pod="openshift-ingress-canary/ingress-canary-c2zhg" Jul 12 18:18:18.281729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4df2da2e-9018-4a69-8776-f152cd0b6052-tmp-dir\") pod \"dns-default-wndcz\" (UID: \"4df2da2e-9018-4a69-8776-f152cd0b6052\") " pod="openshift-dns/dns-default-wndcz" Jul 12 18:18:18.281729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281707 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2-crio-socket\") pod \"insights-runtime-extractor-88mc5\" (UID: \"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2\") " pod="openshift-insights/insights-runtime-extractor-88mc5" Jul 12 18:18:18.281839 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl9pq\" (UniqueName: \"kubernetes.io/projected/b554cd4b-708d-47ab-b6c2-76ae2587ee31-kube-api-access-gl9pq\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.281839 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2-data-volume\") pod \"insights-runtime-extractor-88mc5\" (UID: \"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2\") " pod="openshift-insights/insights-runtime-extractor-88mc5" Jul 12 18:18:18.281930 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281829 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xpm7\" (UniqueName: \"kubernetes.io/projected/d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2-kube-api-access-8xpm7\") pod \"insights-runtime-extractor-88mc5\" (UID: \"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2\") " pod="openshift-insights/insights-runtime-extractor-88mc5" Jul 12 18:18:18.281930 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281858 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4df2da2e-9018-4a69-8776-f152cd0b6052-config-volume\") pod \"dns-default-wndcz\" (UID: \"4df2da2e-9018-4a69-8776-f152cd0b6052\") " pod="openshift-dns/dns-default-wndcz" Jul 12 18:18:18.281930 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b554cd4b-708d-47ab-b6c2-76ae2587ee31-registry-tls\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.282074 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b554cd4b-708d-47ab-b6c2-76ae2587ee31-ca-trust-extracted\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.282074 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b554cd4b-708d-47ab-b6c2-76ae2587ee31-installation-pull-secrets\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.282074 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.281988 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4df2da2e-9018-4a69-8776-f152cd0b6052-tmp-dir\") pod \"dns-default-wndcz\" (UID: \"4df2da2e-9018-4a69-8776-f152cd0b6052\") " pod="openshift-dns/dns-default-wndcz" Jul 12 18:18:18.282074 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.282001 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88mc5\" (UID: \"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2\") " pod="openshift-insights/insights-runtime-extractor-88mc5" Jul 12 18:18:18.285589 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.285571 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4df2da2e-9018-4a69-8776-f152cd0b6052-metrics-tls\") pod \"dns-default-wndcz\" (UID: \"4df2da2e-9018-4a69-8776-f152cd0b6052\") " pod="openshift-dns/dns-default-wndcz" Jul 12 18:18:18.285673 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.285626 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70b7d9dd-771d-4d6c-be24-149e3fa53de0-cert\") pod \"ingress-canary-c2zhg\" (UID: \"70b7d9dd-771d-4d6c-be24-149e3fa53de0\") " pod="openshift-ingress-canary/ingress-canary-c2zhg" Jul 12 18:18:18.288454 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.288433 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:18.291170 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.291153 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l5mw\" (UniqueName: \"kubernetes.io/projected/4df2da2e-9018-4a69-8776-f152cd0b6052-kube-api-access-5l5mw\") pod \"dns-default-wndcz\" (UID: \"4df2da2e-9018-4a69-8776-f152cd0b6052\") " pod="openshift-dns/dns-default-wndcz" Jul 12 18:18:18.291170 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.291171 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59nps\" (UniqueName: \"kubernetes.io/projected/70b7d9dd-771d-4d6c-be24-149e3fa53de0-kube-api-access-59nps\") pod \"ingress-canary-c2zhg\" (UID: \"70b7d9dd-771d-4d6c-be24-149e3fa53de0\") " pod="openshift-ingress-canary/ingress-canary-c2zhg" Jul 12 18:18:18.364597 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.364554 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:18.382904 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.382881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88mc5\" (UID: \"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2\") " pod="openshift-insights/insights-runtime-extractor-88mc5" Jul 12 18:18:18.383009 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.382910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/16e8851a-ed17-47d0-b6d4-f998ea699fc7-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-vths9\" (UID: \"16e8851a-ed17-47d0-b6d4-f998ea699fc7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vths9" Jul 12 18:18:18.383009 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.382934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b554cd4b-708d-47ab-b6c2-76ae2587ee31-bound-sa-token\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.383009 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.382954 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b554cd4b-708d-47ab-b6c2-76ae2587ee31-image-registry-private-configuration\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.383158 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.383075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b554cd4b-708d-47ab-b6c2-76ae2587ee31-registry-certificates\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.383158 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.383117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2-crio-socket\") pod \"insights-runtime-extractor-88mc5\" (UID: \"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2\") " pod="openshift-insights/insights-runtime-extractor-88mc5" Jul 12 18:18:18.383158 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.383147 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl9pq\" (UniqueName: \"kubernetes.io/projected/b554cd4b-708d-47ab-b6c2-76ae2587ee31-kube-api-access-gl9pq\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.383302 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.383170 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2-data-volume\") pod \"insights-runtime-extractor-88mc5\" (UID: \"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2\") " pod="openshift-insights/insights-runtime-extractor-88mc5" Jul 12 18:18:18.383302 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.383196 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xpm7\" (UniqueName: \"kubernetes.io/projected/d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2-kube-api-access-8xpm7\") pod \"insights-runtime-extractor-88mc5\" (UID: \"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2\") " pod="openshift-insights/insights-runtime-extractor-88mc5" Jul 12 18:18:18.383302 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.383238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b554cd4b-708d-47ab-b6c2-76ae2587ee31-registry-tls\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.383302 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.383261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b554cd4b-708d-47ab-b6c2-76ae2587ee31-ca-trust-extracted\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.383302 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.383286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b554cd4b-708d-47ab-b6c2-76ae2587ee31-installation-pull-secrets\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.383540 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.383314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88mc5\" (UID: \"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2\") " pod="openshift-insights/insights-runtime-extractor-88mc5" Jul 12 18:18:18.383540 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.383380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b554cd4b-708d-47ab-b6c2-76ae2587ee31-trusted-ca\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.383540 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.383384 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2-crio-socket\") pod \"insights-runtime-extractor-88mc5\" (UID: \"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2\") " pod="openshift-insights/insights-runtime-extractor-88mc5" Jul 12 18:18:18.383682 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.383614 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88mc5\" (UID: \"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2\") " pod="openshift-insights/insights-runtime-extractor-88mc5" Jul 12 18:18:18.383735 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.383707 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b554cd4b-708d-47ab-b6c2-76ae2587ee31-ca-trust-extracted\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.383896 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.383850 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2-data-volume\") pod \"insights-runtime-extractor-88mc5\" (UID: \"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2\") " pod="openshift-insights/insights-runtime-extractor-88mc5" Jul 12 18:18:18.384084 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.384038 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b554cd4b-708d-47ab-b6c2-76ae2587ee31-registry-certificates\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.384980 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.384959 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b554cd4b-708d-47ab-b6c2-76ae2587ee31-trusted-ca\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.386077 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.386057 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/16e8851a-ed17-47d0-b6d4-f998ea699fc7-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-vths9\" (UID: \"16e8851a-ed17-47d0-b6d4-f998ea699fc7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vths9" Jul 12 18:18:18.386172 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.386083 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b554cd4b-708d-47ab-b6c2-76ae2587ee31-registry-tls\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.386172 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.386098 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b554cd4b-708d-47ab-b6c2-76ae2587ee31-image-registry-private-configuration\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.386172 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.386114 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88mc5\" (UID: \"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2\") " pod="openshift-insights/insights-runtime-extractor-88mc5" Jul 12 18:18:18.386172 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.386137 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b554cd4b-708d-47ab-b6c2-76ae2587ee31-installation-pull-secrets\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.395318 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.395295 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b554cd4b-708d-47ab-b6c2-76ae2587ee31-bound-sa-token\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.395538 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.395521 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl9pq\" (UniqueName: \"kubernetes.io/projected/b554cd4b-708d-47ab-b6c2-76ae2587ee31-kube-api-access-gl9pq\") pod \"image-registry-5f788c444b-9sjcx\" (UID: \"b554cd4b-708d-47ab-b6c2-76ae2587ee31\") " pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.396466 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.396445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xpm7\" (UniqueName: \"kubernetes.io/projected/d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2-kube-api-access-8xpm7\") pod \"insights-runtime-extractor-88mc5\" (UID: \"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2\") " pod="openshift-insights/insights-runtime-extractor-88mc5" Jul 12 18:18:18.462396 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.462315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wndcz" Jul 12 18:18:18.464649 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.464608 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:18.468250 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.468231 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c2zhg" Jul 12 18:18:18.494277 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.494247 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-88mc5" Jul 12 18:18:18.562272 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.561934 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vths9" Jul 12 18:18:18.567215 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.567194 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:18.631717 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.631688 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c2zhg"] Jul 12 18:18:18.637613 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:18:18.637580 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b7d9dd_771d_4d6c_be24_149e3fa53de0.slice/crio-e462b59356c2528a8b2ca0ae8a794b2f132db6fa92207d58bd66903eb4c0c767 WatchSource:0}: Error finding container e462b59356c2528a8b2ca0ae8a794b2f132db6fa92207d58bd66903eb4c0c767: Status 404 returned error can't find the container with id e462b59356c2528a8b2ca0ae8a794b2f132db6fa92207d58bd66903eb4c0c767 Jul 12 18:18:18.638225 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.638150 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wndcz"] Jul 12 18:18:18.658206 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.658176 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-88mc5"] Jul 12 18:18:18.714584 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.714518 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f788c444b-9sjcx"] Jul 12 18:18:18.717217 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:18:18.717190 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb554cd4b_708d_47ab_b6c2_76ae2587ee31.slice/crio-197843633b3942229cdadbf9098e3ac39858a86412c69bcd5ccfbd48fb26f5aa WatchSource:0}: Error finding container 197843633b3942229cdadbf9098e3ac39858a86412c69bcd5ccfbd48fb26f5aa: Status 404 returned error can't find the container with id 197843633b3942229cdadbf9098e3ac39858a86412c69bcd5ccfbd48fb26f5aa Jul 12 18:18:18.727412 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:18.727391 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vths9"] Jul 12 18:18:18.729353 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:18:18.729331 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e8851a_ed17_47d0_b6d4_f998ea699fc7.slice/crio-b44c3ee7a7c6a8710c78da5992ab500f2ba28bfe661c4309c3041e991a4e3b02 WatchSource:0}: Error finding container b44c3ee7a7c6a8710c78da5992ab500f2ba28bfe661c4309c3041e991a4e3b02: Status 404 returned error can't find the container with id b44c3ee7a7c6a8710c78da5992ab500f2ba28bfe661c4309c3041e991a4e3b02 Jul 12 18:18:19.041422 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:19.041347 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vths9" event={"ID":"16e8851a-ed17-47d0-b6d4-f998ea699fc7","Type":"ContainerStarted","Data":"b44c3ee7a7c6a8710c78da5992ab500f2ba28bfe661c4309c3041e991a4e3b02"} Jul 12 18:18:19.042935 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:19.042905 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" event={"ID":"b554cd4b-708d-47ab-b6c2-76ae2587ee31","Type":"ContainerStarted","Data":"36a41677c2c9547442bd4cafc3417cc8209ff3e77cbb31be9f7a0826c03ce30b"} Jul 12 18:18:19.043074 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:19.042950 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" event={"ID":"b554cd4b-708d-47ab-b6c2-76ae2587ee31","Type":"ContainerStarted","Data":"197843633b3942229cdadbf9098e3ac39858a86412c69bcd5ccfbd48fb26f5aa"} Jul 12 18:18:19.043074 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:19.043001 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:19.044167 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:19.044141 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88mc5" event={"ID":"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2","Type":"ContainerStarted","Data":"4c6e332dee3d8e3e825245724c30bb3d8d63efa3edb09947dcc5c6fa58028922"} Jul 12 18:18:19.045250 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:19.045228 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c2zhg" event={"ID":"70b7d9dd-771d-4d6c-be24-149e3fa53de0","Type":"ContainerStarted","Data":"e462b59356c2528a8b2ca0ae8a794b2f132db6fa92207d58bd66903eb4c0c767"} Jul 12 18:18:19.046305 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:19.046285 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wndcz" event={"ID":"4df2da2e-9018-4a69-8776-f152cd0b6052","Type":"ContainerStarted","Data":"1b70519a997f094abb609a00df7284d761fcf04ac9f70ca7e76fa7a888f23248"} Jul 12 18:18:19.062710 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:19.062173 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" podStartSLOduration=1.062158248 podStartE2EDuration="1.062158248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:18:19.061735315 +0000 UTC m=+60.829032912" watchObservedRunningTime="2026-07-12 18:18:19.062158248 +0000 UTC m=+60.829455840" Jul 12 18:18:19.993051 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:19.993017 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pq4vs" Jul 12 18:18:20.103228 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:20.103196 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:20.153246 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:20.153212 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:22.379680 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:22.379632 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:22.429175 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:22.429095 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:24.064451 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.064105 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vths9" event={"ID":"16e8851a-ed17-47d0-b6d4-f998ea699fc7","Type":"ContainerStarted","Data":"2ccb326a32ae559298e6c429776112a7b1de839d74e1e7732b2237f6ae7bff89"} Jul 12 18:18:24.064451 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.064355 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vths9" Jul 12 18:18:24.068551 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.068517 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c2zhg" event={"ID":"70b7d9dd-771d-4d6c-be24-149e3fa53de0","Type":"ContainerStarted","Data":"59d8a6440141c9126fcce6bd1cbfaa59bdbee0ac7358e081e79deb0ef2ca027f"} Jul 12 18:18:24.070756 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.070728 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wndcz" event={"ID":"4df2da2e-9018-4a69-8776-f152cd0b6052","Type":"ContainerStarted","Data":"d3c98d0645de54a165072d4481805cb505c0b39a23372c81053addadd7df835c"} Jul 12 18:18:24.071534 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.071515 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vths9" Jul 12 18:18:24.078711 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.078547 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vths9" podStartSLOduration=0.958240197 podStartE2EDuration="6.078530299s" totalImagesPullingTime="5.120290102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:18:24.078514221 +0000 UTC m=+65.845811805" watchObservedRunningTime="2026-07-12 18:18:24.078530299 +0000 UTC m=+65.845827883" Jul 12 18:18:24.093843 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.093775 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-c2zhg" podStartSLOduration=0.857491077 podStartE2EDuration="6.093756882s" totalImagesPullingTime="5.236265805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:18:24.093644684 +0000 UTC m=+65.860942264" watchObservedRunningTime="2026-07-12 18:18:24.093756882 +0000 UTC m=+65.861054442" Jul 12 18:18:24.442025 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.441937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs\") pod \"network-metrics-daemon-jw4j5\" (UID: \"6a42ac1b-6e6a-4a60-8d10-42c2765b38ec\") " pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:18:24.444379 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.444361 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 12 18:18:24.454500 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.454472 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a42ac1b-6e6a-4a60-8d10-42c2765b38ec-metrics-certs\") pod \"network-metrics-daemon-jw4j5\" (UID: \"6a42ac1b-6e6a-4a60-8d10-42c2765b38ec\") " pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:18:24.530343 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.530307 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p9cbp\"" Jul 12 18:18:24.538323 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.538301 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw4j5" Jul 12 18:18:24.542704 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.542685 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret\") pod \"global-pull-secret-syncer-pjl7r\" (UID: \"8a8808fa-07dc-4568-b10f-0a84be341957\") " pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:18:24.542765 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.542736 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nsx7\" (UniqueName: \"kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7\") pod \"network-check-target-g7w97\" (UID: \"62fe9c7a-2477-4d68-b119-2dcc30e1c944\") " pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:18:24.545119 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.545101 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 12 18:18:24.545192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.545176 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 12 18:18:24.555202 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.555179 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 12 18:18:24.555313 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.555263 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a8808fa-07dc-4568-b10f-0a84be341957-original-pull-secret\") pod \"global-pull-secret-syncer-pjl7r\" (UID: \"8a8808fa-07dc-4568-b10f-0a84be341957\") " pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:18:24.565403 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.565379 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nsx7\" (UniqueName: \"kubernetes.io/projected/62fe9c7a-2477-4d68-b119-2dcc30e1c944-kube-api-access-2nsx7\") pod \"network-check-target-g7w97\" (UID: \"62fe9c7a-2477-4d68-b119-2dcc30e1c944\") " pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:18:24.689875 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.689843 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jw4j5"] Jul 12 18:18:24.692619 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:18:24.692552 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a42ac1b_6e6a_4a60_8d10_42c2765b38ec.slice/crio-1e5a32e07977267107d37acc470c1effd0a807515a47be30e1ec6b2bdde374b4 WatchSource:0}: Error finding container 1e5a32e07977267107d37acc470c1effd0a807515a47be30e1ec6b2bdde374b4: Status 404 returned error can't find the container with id 1e5a32e07977267107d37acc470c1effd0a807515a47be30e1ec6b2bdde374b4 Jul 12 18:18:24.758666 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.758633 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:24.792559 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.790895 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-pwrnr"] Jul 12 18:18:24.821994 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.821953 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjl7r" Jul 12 18:18:24.824028 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.824003 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" Jul 12 18:18:24.826715 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.826684 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 12 18:18:24.826851 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.826729 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 12 18:18:24.826851 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.826684 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 12 18:18:24.826851 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.826773 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wrmnk\"" Jul 12 18:18:24.826851 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.826689 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 12 18:18:24.827070 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.827055 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 12 18:18:24.827648 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.827632 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-pwrnr"] Jul 12 18:18:24.833983 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.833965 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hb5mg\"" Jul 12 18:18:24.842102 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.842086 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:18:24.842352 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.842330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:24.925544 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.925497 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:24.946275 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.946159 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eec26ff9-0bdb-42f3-ab2b-4deee347a4d9-metrics-client-ca\") pod \"prometheus-operator-64785b6675-pwrnr\" (UID: \"eec26ff9-0bdb-42f3-ab2b-4deee347a4d9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" Jul 12 18:18:24.946275 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.946216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/eec26ff9-0bdb-42f3-ab2b-4deee347a4d9-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-pwrnr\" (UID: \"eec26ff9-0bdb-42f3-ab2b-4deee347a4d9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" Jul 12 18:18:24.946445 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.946278 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6bql\" (UniqueName: \"kubernetes.io/projected/eec26ff9-0bdb-42f3-ab2b-4deee347a4d9-kube-api-access-f6bql\") pod \"prometheus-operator-64785b6675-pwrnr\" (UID: \"eec26ff9-0bdb-42f3-ab2b-4deee347a4d9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" Jul 12 18:18:24.946445 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.946309 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eec26ff9-0bdb-42f3-ab2b-4deee347a4d9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-pwrnr\" (UID: \"eec26ff9-0bdb-42f3-ab2b-4deee347a4d9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" Jul 12 18:18:24.967682 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:24.967656 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pjl7r"] Jul 12 18:18:24.971314 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:18:24.971275 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8808fa_07dc_4568_b10f_0a84be341957.slice/crio-2ef3b2fb202627bd3fed24ed3e32c372beb37b6bff6464a82d0ccd5f36fd80c5 WatchSource:0}: Error finding container 2ef3b2fb202627bd3fed24ed3e32c372beb37b6bff6464a82d0ccd5f36fd80c5: Status 404 returned error can't find the container with id 2ef3b2fb202627bd3fed24ed3e32c372beb37b6bff6464a82d0ccd5f36fd80c5 Jul 12 18:18:25.011986 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.011833 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g7w97"] Jul 12 18:18:25.014371 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:18:25.014340 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fe9c7a_2477_4d68_b119_2dcc30e1c944.slice/crio-266473877f800e638aa9be2f6849a8aca7cc07f80b3eccddc2668c10ccc0fdc9 WatchSource:0}: Error finding container 266473877f800e638aa9be2f6849a8aca7cc07f80b3eccddc2668c10ccc0fdc9: Status 404 returned error can't find the container with id 266473877f800e638aa9be2f6849a8aca7cc07f80b3eccddc2668c10ccc0fdc9 Jul 12 18:18:25.047700 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.047669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eec26ff9-0bdb-42f3-ab2b-4deee347a4d9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-pwrnr\" (UID: \"eec26ff9-0bdb-42f3-ab2b-4deee347a4d9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" Jul 12 18:18:25.047868 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.047726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eec26ff9-0bdb-42f3-ab2b-4deee347a4d9-metrics-client-ca\") pod \"prometheus-operator-64785b6675-pwrnr\" (UID: \"eec26ff9-0bdb-42f3-ab2b-4deee347a4d9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" Jul 12 18:18:25.047868 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.047747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/eec26ff9-0bdb-42f3-ab2b-4deee347a4d9-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-pwrnr\" (UID: \"eec26ff9-0bdb-42f3-ab2b-4deee347a4d9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" Jul 12 18:18:25.047868 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.047770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6bql\" (UniqueName: \"kubernetes.io/projected/eec26ff9-0bdb-42f3-ab2b-4deee347a4d9-kube-api-access-f6bql\") pod \"prometheus-operator-64785b6675-pwrnr\" (UID: \"eec26ff9-0bdb-42f3-ab2b-4deee347a4d9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" Jul 12 18:18:25.048622 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.048593 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eec26ff9-0bdb-42f3-ab2b-4deee347a4d9-metrics-client-ca\") pod \"prometheus-operator-64785b6675-pwrnr\" (UID: \"eec26ff9-0bdb-42f3-ab2b-4deee347a4d9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" Jul 12 18:18:25.050165 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.050145 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eec26ff9-0bdb-42f3-ab2b-4deee347a4d9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-pwrnr\" (UID: \"eec26ff9-0bdb-42f3-ab2b-4deee347a4d9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" Jul 12 18:18:25.050340 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.050318 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/eec26ff9-0bdb-42f3-ab2b-4deee347a4d9-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-pwrnr\" (UID: \"eec26ff9-0bdb-42f3-ab2b-4deee347a4d9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" Jul 12 18:18:25.056351 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.056326 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6bql\" (UniqueName: \"kubernetes.io/projected/eec26ff9-0bdb-42f3-ab2b-4deee347a4d9-kube-api-access-f6bql\") pod \"prometheus-operator-64785b6675-pwrnr\" (UID: \"eec26ff9-0bdb-42f3-ab2b-4deee347a4d9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" Jul 12 18:18:25.076077 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.076038 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g7w97" event={"ID":"62fe9c7a-2477-4d68-b119-2dcc30e1c944","Type":"ContainerStarted","Data":"266473877f800e638aa9be2f6849a8aca7cc07f80b3eccddc2668c10ccc0fdc9"} Jul 12 18:18:25.077358 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.077328 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pjl7r" event={"ID":"8a8808fa-07dc-4568-b10f-0a84be341957","Type":"ContainerStarted","Data":"2ef3b2fb202627bd3fed24ed3e32c372beb37b6bff6464a82d0ccd5f36fd80c5"} Jul 12 18:18:25.078892 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.078868 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88mc5" event={"ID":"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2","Type":"ContainerStarted","Data":"a992d814111de1b7332ce533e64104d9b1ef7cca47b0f93fc0782ecf579b6ed1"} Jul 12 18:18:25.080840 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.080814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wndcz" event={"ID":"4df2da2e-9018-4a69-8776-f152cd0b6052","Type":"ContainerStarted","Data":"3039dcb0cce67747e255dff368a5e463296f4952752e17becab3ee24209abdc7"} Jul 12 18:18:25.080920 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.080907 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wndcz" Jul 12 18:18:25.082209 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.082185 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jw4j5" event={"ID":"6a42ac1b-6e6a-4a60-8d10-42c2765b38ec","Type":"ContainerStarted","Data":"1e5a32e07977267107d37acc470c1effd0a807515a47be30e1ec6b2bdde374b4"} Jul 12 18:18:25.098056 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.098006 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wndcz" podStartSLOduration=1.877819556 podStartE2EDuration="7.097985386s" totalImagesPullingTime="5.22016583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:18:25.097547698 +0000 UTC m=+66.864845279" watchObservedRunningTime="2026-07-12 18:18:25.097985386 +0000 UTC m=+66.865282969" Jul 12 18:18:25.132973 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.132915 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" Jul 12 18:18:25.306256 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:25.306230 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-pwrnr"] Jul 12 18:18:25.309858 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:18:25.309829 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec26ff9_0bdb_42f3_ab2b_4deee347a4d9.slice/crio-2d617e66f555622418c7d1096895ffcd75c6133a80abe7db1a2ba5766e2267a4 WatchSource:0}: Error finding container 2d617e66f555622418c7d1096895ffcd75c6133a80abe7db1a2ba5766e2267a4: Status 404 returned error can't find the container with id 2d617e66f555622418c7d1096895ffcd75c6133a80abe7db1a2ba5766e2267a4 Jul 12 18:18:26.090056 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:26.090014 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88mc5" event={"ID":"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2","Type":"ContainerStarted","Data":"a96150b19245169b9a1d7f5cf70f7feb7d2b72bf06cf9a76e65ac9f262de5e84"} Jul 12 18:18:26.092247 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:26.091834 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jw4j5" event={"ID":"6a42ac1b-6e6a-4a60-8d10-42c2765b38ec","Type":"ContainerStarted","Data":"ebff9b5b382fc42b4f0d3e4ce680def940fe00fe7d3148027b92f88949ca96e0"} Jul 12 18:18:26.093157 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:26.092689 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jw4j5" event={"ID":"6a42ac1b-6e6a-4a60-8d10-42c2765b38ec","Type":"ContainerStarted","Data":"8293d001036ebf91f7f20be8b29f95edb5a922440ce731665e754029147a9662"} Jul 12 18:18:26.096364 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:26.095056 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" event={"ID":"eec26ff9-0bdb-42f3-ab2b-4deee347a4d9","Type":"ContainerStarted","Data":"2d617e66f555622418c7d1096895ffcd75c6133a80abe7db1a2ba5766e2267a4"} Jul 12 18:18:26.108381 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:26.107950 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jw4j5" podStartSLOduration=66.120839708 podStartE2EDuration="1m7.107929183s" totalImagesPullingTime="987.089475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:18:26.106913751 +0000 UTC m=+67.874211334" watchObservedRunningTime="2026-07-12 18:18:26.107929183 +0000 UTC m=+67.875226767" Jul 12 18:18:26.222300 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:26.222222 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:26.274600 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:26.274561 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:27.105477 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:27.105346 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" event={"ID":"eec26ff9-0bdb-42f3-ab2b-4deee347a4d9","Type":"ContainerStarted","Data":"79b314dcd8b4081849a54a41244daf4ccb57623a3ea4c3f40a6065463353686e"} Jul 12 18:18:27.306621 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:27.306530 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:28.110535 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:28.110496 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" event={"ID":"eec26ff9-0bdb-42f3-ab2b-4deee347a4d9","Type":"ContainerStarted","Data":"5aaa8ae7cea02e9ba17510abd016bfcb64c3a6101c96345c2fbd3e7597fc7523"} Jul 12 18:18:28.128496 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:28.128448 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-pwrnr" podStartSLOduration=2.431572437 podStartE2EDuration="4.128432802s" totalImagesPullingTime="1.696860365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:18:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:18:28.127132879 +0000 UTC m=+69.894430463" watchObservedRunningTime="2026-07-12 18:18:28.128432802 +0000 UTC m=+69.895730426" Jul 12 18:18:29.777272 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:29.777230 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-zldlk"] Jul 12 18:18:29.781529 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:29.781507 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-zldlk" Jul 12 18:18:29.784041 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:29.784017 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 12 18:18:29.784157 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:29.784079 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-p6gbh\"" Jul 12 18:18:29.784305 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:29.784287 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 12 18:18:29.793442 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:29.790680 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-zldlk"] Jul 12 18:18:29.796262 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:29.796233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:29.886150 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:29.886114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77pqj\" (UniqueName: \"kubernetes.io/projected/c6483f3c-7c4a-48d4-90f8-ff4deb9aa0a4-kube-api-access-77pqj\") pod \"downloads-68c9cbb497-zldlk\" (UID: \"c6483f3c-7c4a-48d4-90f8-ff4deb9aa0a4\") " pod="openshift-console/downloads-68c9cbb497-zldlk" Jul 12 18:18:29.898688 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:29.898639 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:29.987488 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:29.987445 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77pqj\" (UniqueName: \"kubernetes.io/projected/c6483f3c-7c4a-48d4-90f8-ff4deb9aa0a4-kube-api-access-77pqj\") pod \"downloads-68c9cbb497-zldlk\" (UID: \"c6483f3c-7c4a-48d4-90f8-ff4deb9aa0a4\") " pod="openshift-console/downloads-68c9cbb497-zldlk" Jul 12 18:18:29.996667 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:29.996640 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77pqj\" (UniqueName: \"kubernetes.io/projected/c6483f3c-7c4a-48d4-90f8-ff4deb9aa0a4-kube-api-access-77pqj\") pod \"downloads-68c9cbb497-zldlk\" (UID: \"c6483f3c-7c4a-48d4-90f8-ff4deb9aa0a4\") " pod="openshift-console/downloads-68c9cbb497-zldlk" Jul 12 18:18:30.000530 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.000471 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:30.096759 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.096718 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-zldlk" Jul 12 18:18:30.104629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.104595 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:30.171709 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.171668 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b"] Jul 12 18:18:30.175081 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.175052 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" Jul 12 18:18:30.177600 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.177576 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 12 18:18:30.177722 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.177577 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bkg8b\"" Jul 12 18:18:30.177811 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.177747 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 12 18:18:30.184845 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.184815 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b"] Jul 12 18:18:30.191395 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.191368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:30.191697 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.191674 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vpp5s"] Jul 12 18:18:30.195059 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.195038 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.197622 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.197498 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 12 18:18:30.197622 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.197510 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 12 18:18:30.197622 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.197505 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 12 18:18:30.197879 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.197844 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ss2pq\"" Jul 12 18:18:30.272202 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.272167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:30.289922 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.289888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-sys\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.290095 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.289939 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.290095 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.290002 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-node-exporter-tls\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.290095 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.290028 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-node-exporter-wtmp\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.290095 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.290062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-node-exporter-textfile\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.290095 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.290081 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12e428fb-eb05-4442-90c5-183a117de7d5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-xmj8b\" (UID: \"12e428fb-eb05-4442-90c5-183a117de7d5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" Jul 12 18:18:30.290293 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.290103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12e428fb-eb05-4442-90c5-183a117de7d5-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-xmj8b\" (UID: \"12e428fb-eb05-4442-90c5-183a117de7d5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" Jul 12 18:18:30.290293 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.290207 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12e428fb-eb05-4442-90c5-183a117de7d5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-xmj8b\" (UID: \"12e428fb-eb05-4442-90c5-183a117de7d5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" Jul 12 18:18:30.290293 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.290253 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-metrics-client-ca\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.290422 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.290290 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhh9z\" (UniqueName: \"kubernetes.io/projected/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-kube-api-access-hhh9z\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.290422 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.290321 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk87k\" (UniqueName: \"kubernetes.io/projected/12e428fb-eb05-4442-90c5-183a117de7d5-kube-api-access-hk87k\") pod \"openshift-state-metrics-84bd68b896-xmj8b\" (UID: \"12e428fb-eb05-4442-90c5-183a117de7d5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" Jul 12 18:18:30.290422 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.290352 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-root\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.290422 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.290400 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-node-exporter-accelerators-collector-config\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.372325 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.372233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:30.391501 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.391458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-metrics-client-ca\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.391501 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.391508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhh9z\" (UniqueName: \"kubernetes.io/projected/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-kube-api-access-hhh9z\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.391722 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.391538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk87k\" (UniqueName: \"kubernetes.io/projected/12e428fb-eb05-4442-90c5-183a117de7d5-kube-api-access-hk87k\") pod \"openshift-state-metrics-84bd68b896-xmj8b\" (UID: \"12e428fb-eb05-4442-90c5-183a117de7d5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" Jul 12 18:18:30.391722 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.391557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-root\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.391722 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.391587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-node-exporter-accelerators-collector-config\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.391722 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.391676 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-root\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.391722 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.391710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-sys\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.391951 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.391754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.391951 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.391777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-node-exporter-tls\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.391951 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.391827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-node-exporter-wtmp\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.391951 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.391860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-node-exporter-textfile\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.391951 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.391892 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12e428fb-eb05-4442-90c5-183a117de7d5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-xmj8b\" (UID: \"12e428fb-eb05-4442-90c5-183a117de7d5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" Jul 12 18:18:30.391951 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.391917 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12e428fb-eb05-4442-90c5-183a117de7d5-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-xmj8b\" (UID: \"12e428fb-eb05-4442-90c5-183a117de7d5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" Jul 12 18:18:30.391951 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.391941 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12e428fb-eb05-4442-90c5-183a117de7d5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-xmj8b\" (UID: \"12e428fb-eb05-4442-90c5-183a117de7d5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" Jul 12 18:18:30.392296 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:18:30.392041 2576 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 12 18:18:30.392296 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:18:30.392097 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e428fb-eb05-4442-90c5-183a117de7d5-openshift-state-metrics-tls podName:12e428fb-eb05-4442-90c5-183a117de7d5 nodeName:}" failed. No retries permitted until 2026-07-12 18:18:30.892077234 +0000 UTC m=+72.659374808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/12e428fb-eb05-4442-90c5-183a117de7d5-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-xmj8b" (UID: "12e428fb-eb05-4442-90c5-183a117de7d5") : secret "openshift-state-metrics-tls" not found Jul 12 18:18:30.392296 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.392150 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-metrics-client-ca\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.392296 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.392159 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-node-exporter-wtmp\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.392296 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.392215 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-sys\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.392529 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.392452 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-node-exporter-accelerators-collector-config\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.392740 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.392706 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-node-exporter-textfile\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.392976 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.392937 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12e428fb-eb05-4442-90c5-183a117de7d5-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-xmj8b\" (UID: \"12e428fb-eb05-4442-90c5-183a117de7d5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" Jul 12 18:18:30.394780 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.394754 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.394950 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.394903 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-node-exporter-tls\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.395554 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.395089 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12e428fb-eb05-4442-90c5-183a117de7d5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-xmj8b\" (UID: \"12e428fb-eb05-4442-90c5-183a117de7d5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" Jul 12 18:18:30.400880 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.400833 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk87k\" (UniqueName: \"kubernetes.io/projected/12e428fb-eb05-4442-90c5-183a117de7d5-kube-api-access-hk87k\") pod \"openshift-state-metrics-84bd68b896-xmj8b\" (UID: \"12e428fb-eb05-4442-90c5-183a117de7d5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" Jul 12 18:18:30.401931 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.401906 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhh9z\" (UniqueName: \"kubernetes.io/projected/9cbfbdb1-45ff-461a-9517-6174e7ce7dfa-kube-api-access-hhh9z\") pod \"node-exporter-vpp5s\" (UID: \"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa\") " pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.472445 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.472390 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:30.505740 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.505692 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vpp5s" Jul 12 18:18:30.572625 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.572588 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:30.672885 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.672767 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:30.845695 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:18:30.845660 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cbfbdb1_45ff_461a_9517_6174e7ce7dfa.slice/crio-1f25b626df3f429158ef0f8d9814fd3cdd4983291309722bcf604bd171751496 WatchSource:0}: Error finding container 1f25b626df3f429158ef0f8d9814fd3cdd4983291309722bcf604bd171751496: Status 404 returned error can't find the container with id 1f25b626df3f429158ef0f8d9814fd3cdd4983291309722bcf604bd171751496 Jul 12 18:18:30.897602 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.896893 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12e428fb-eb05-4442-90c5-183a117de7d5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-xmj8b\" (UID: \"12e428fb-eb05-4442-90c5-183a117de7d5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" Jul 12 18:18:30.897602 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:18:30.897039 2576 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 12 18:18:30.897602 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:18:30.897120 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e428fb-eb05-4442-90c5-183a117de7d5-openshift-state-metrics-tls podName:12e428fb-eb05-4442-90c5-183a117de7d5 nodeName:}" failed. No retries permitted until 2026-07-12 18:18:31.897099137 +0000 UTC m=+73.664396719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/12e428fb-eb05-4442-90c5-183a117de7d5-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-xmj8b" (UID: "12e428fb-eb05-4442-90c5-183a117de7d5") : secret "openshift-state-metrics-tls" not found Jul 12 18:18:30.913660 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.913581 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:30.980298 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.980262 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-zldlk"] Jul 12 18:18:30.983131 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:30.983063 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:30.984864 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:18:30.984828 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6483f3c_7c4a_48d4_90f8_ff4deb9aa0a4.slice/crio-af7534c2f5c85bc3985d9a4802df2f957c3e6bda639d220b9601fe839821e85c WatchSource:0}: Error finding container af7534c2f5c85bc3985d9a4802df2f957c3e6bda639d220b9601fe839821e85c: Status 404 returned error can't find the container with id af7534c2f5c85bc3985d9a4802df2f957c3e6bda639d220b9601fe839821e85c Jul 12 18:18:31.124532 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.124495 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vpp5s" event={"ID":"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa","Type":"ContainerStarted","Data":"1f25b626df3f429158ef0f8d9814fd3cdd4983291309722bcf604bd171751496"} Jul 12 18:18:31.125837 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.125810 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-zldlk" event={"ID":"c6483f3c-7c4a-48d4-90f8-ff4deb9aa0a4","Type":"ContainerStarted","Data":"af7534c2f5c85bc3985d9a4802df2f957c3e6bda639d220b9601fe839821e85c"} Jul 12 18:18:31.127392 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.127364 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g7w97" event={"ID":"62fe9c7a-2477-4d68-b119-2dcc30e1c944","Type":"ContainerStarted","Data":"6545686b6620b260fb28b66c1ba85bf03a3557327f6993f56873a440ab3e7986"} Jul 12 18:18:31.127558 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.127520 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:18:31.129232 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.129209 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pjl7r" event={"ID":"8a8808fa-07dc-4568-b10f-0a84be341957","Type":"ContainerStarted","Data":"dfac4bce5ef3c06c1c0063127e984fbc3e9fbb103e452e38e833ae18830d7aba"} Jul 12 18:18:31.131341 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.131323 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88mc5" event={"ID":"d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2","Type":"ContainerStarted","Data":"e6f407f143de777cc156e56c721a670c4683b76d007446ff42dfe359e172127f"} Jul 12 18:18:31.145118 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.145078 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g7w97" podStartSLOduration=66.315243177 podStartE2EDuration="1m12.145067038s" totalImagesPullingTime="5.829823861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:18:31.143994136 +0000 UTC m=+72.911291718" watchObservedRunningTime="2026-07-12 18:18:31.145067038 +0000 UTC m=+72.912364646" Jul 12 18:18:31.159722 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.159674 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pjl7r" podStartSLOduration=66.281607012 podStartE2EDuration="1m12.159656866s" totalImagesPullingTime="5.878049854s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 18:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:18:31.157935404 +0000 UTC m=+72.925232987" watchObservedRunningTime="2026-07-12 18:18:31.159656866 +0000 UTC m=+72.926954453" Jul 12 18:18:31.164911 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.164881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:31.176171 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.176133 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-88mc5" podStartSLOduration=1.170601163 podStartE2EDuration="13.176119354s" totalImagesPullingTime="12.005518191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 18:18:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:18:31.174914191 +0000 UTC m=+72.942211775" watchObservedRunningTime="2026-07-12 18:18:31.176119354 +0000 UTC m=+72.943416936" Jul 12 18:18:31.258294 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.258156 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 12 18:18:31.258294 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.258229 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:31.263302 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.262328 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.264694 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.264674 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 12 18:18:31.264973 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.264954 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 12 18:18:31.265189 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.265173 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 12 18:18:31.265743 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.265479 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 12 18:18:31.265743 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.265510 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 12 18:18:31.265743 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.265533 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 12 18:18:31.265743 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.265549 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 12 18:18:31.265743 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.265483 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 12 18:18:31.265743 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.265662 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 12 18:18:31.265743 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.265748 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rn9kx\"" Jul 12 18:18:31.273471 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.273453 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 12 18:18:31.300434 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.300402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8b090ef-4073-4a60-b30b-c54e39cff6df-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.300595 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.300441 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.300595 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.300521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.300595 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.300563 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b090ef-4073-4a60-b30b-c54e39cff6df-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.300775 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.300651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.300775 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.300683 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-web-config\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.300775 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.300712 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kghhb\" (UniqueName: \"kubernetes.io/projected/b8b090ef-4073-4a60-b30b-c54e39cff6df-kube-api-access-kghhb\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.300955 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.300775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.300955 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.300869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b090ef-4073-4a60-b30b-c54e39cff6df-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.300955 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.300897 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b8b090ef-4073-4a60-b30b-c54e39cff6df-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.300955 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.300928 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-config-volume\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.300955 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.300952 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8b090ef-4073-4a60-b30b-c54e39cff6df-config-out\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.301249 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.300977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.323646 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.323593 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:31.379150 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.379106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:31.401520 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.401486 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.401696 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.401547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b090ef-4073-4a60-b30b-c54e39cff6df-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.401696 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:18:31.401686 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8b090ef-4073-4a60-b30b-c54e39cff6df-alertmanager-trusted-ca-bundle podName:b8b090ef-4073-4a60-b30b-c54e39cff6df nodeName:}" failed. No retries permitted until 2026-07-12 18:18:31.901662709 +0000 UTC m=+73.668960283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/b8b090ef-4073-4a60-b30b-c54e39cff6df-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "b8b090ef-4073-4a60-b30b-c54e39cff6df") : configmap references non-existent config key: ca-bundle.crt Jul 12 18:18:31.401844 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.401709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b8b090ef-4073-4a60-b30b-c54e39cff6df-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.401844 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.401743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-config-volume\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.401844 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.401770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8b090ef-4073-4a60-b30b-c54e39cff6df-config-out\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.401844 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.401813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.402042 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.401861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8b090ef-4073-4a60-b30b-c54e39cff6df-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.402042 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.401893 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.402042 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.401945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.402042 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.401977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b090ef-4073-4a60-b30b-c54e39cff6df-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.402042 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.402014 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.402042 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.402040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-web-config\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.402353 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.402069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kghhb\" (UniqueName: \"kubernetes.io/projected/b8b090ef-4073-4a60-b30b-c54e39cff6df-kube-api-access-kghhb\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.402353 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.402137 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b8b090ef-4073-4a60-b30b-c54e39cff6df-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.402353 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:18:31.402242 2576 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 12 18:18:31.402353 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:18:31.402289 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-main-tls podName:b8b090ef-4073-4a60-b30b-c54e39cff6df nodeName:}" failed. No retries permitted until 2026-07-12 18:18:31.902273545 +0000 UTC m=+73.669571111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "b8b090ef-4073-4a60-b30b-c54e39cff6df") : secret "alertmanager-main-tls" not found Jul 12 18:18:31.402985 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.402919 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b090ef-4073-4a60-b30b-c54e39cff6df-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.404929 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.404908 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-config-volume\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.405237 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.405210 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.405444 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.405425 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.405645 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.405618 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8b090ef-4073-4a60-b30b-c54e39cff6df-config-out\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.405709 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.405660 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8b090ef-4073-4a60-b30b-c54e39cff6df-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.405821 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.405800 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-web-config\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.406275 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.406254 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.407604 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.407585 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.412193 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.412172 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kghhb\" (UniqueName: \"kubernetes.io/projected/b8b090ef-4073-4a60-b30b-c54e39cff6df-kube-api-access-kghhb\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.537032 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.536948 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:31.679312 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.679281 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:31.780246 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.780208 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:31.879995 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.879952 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:31.906921 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.906890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.907088 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.906975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b090ef-4073-4a60-b30b-c54e39cff6df-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.907088 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.907050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12e428fb-eb05-4442-90c5-183a117de7d5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-xmj8b\" (UID: \"12e428fb-eb05-4442-90c5-183a117de7d5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" Jul 12 18:18:31.907934 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.907911 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b090ef-4073-4a60-b30b-c54e39cff6df-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.909692 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.909666 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:31.909774 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.909725 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12e428fb-eb05-4442-90c5-183a117de7d5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-xmj8b\" (UID: \"12e428fb-eb05-4442-90c5-183a117de7d5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" Jul 12 18:18:31.988768 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:31.988730 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" Jul 12 18:18:32.136952 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.136918 2576 generic.go:356] "Generic (PLEG): container finished" podID="9cbfbdb1-45ff-461a-9517-6174e7ce7dfa" containerID="e15d42c563b9ba18e1ef558f57040be7d808a38b54723572e40cc7c910a88ee2" exitCode=0 Jul 12 18:18:32.137150 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.137016 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vpp5s" event={"ID":"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa","Type":"ContainerDied","Data":"e15d42c563b9ba18e1ef558f57040be7d808a38b54723572e40cc7c910a88ee2"} Jul 12 18:18:32.141743 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.141689 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b"] Jul 12 18:18:32.144669 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:18:32.144634 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e428fb_eb05_4442_90c5_183a117de7d5.slice/crio-5cc45ae153cbbbb645d9a3cee5372d7c230801fd8a8599fff3d8af1d72031c50 WatchSource:0}: Error finding container 5cc45ae153cbbbb645d9a3cee5372d7c230801fd8a8599fff3d8af1d72031c50: Status 404 returned error can't find the container with id 5cc45ae153cbbbb645d9a3cee5372d7c230801fd8a8599fff3d8af1d72031c50 Jul 12 18:18:32.159579 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.159554 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6"] Jul 12 18:18:32.165510 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.165485 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.168062 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.167901 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 12 18:18:32.170035 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.168376 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 12 18:18:32.170035 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.168396 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 12 18:18:32.170035 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.168626 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5ppf9\"" Jul 12 18:18:32.170035 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.168817 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 12 18:18:32.170035 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.169677 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 12 18:18:32.170035 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.169899 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8ru3ouvjkdka5\"" Jul 12 18:18:32.175728 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.175088 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6"] Jul 12 18:18:32.175728 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.175176 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:18:32.179187 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.179100 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:32.209860 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.209823 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.209860 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.209858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9dnh\" (UniqueName: \"kubernetes.io/projected/f8c24501-1c77-4804-b703-5ae46ee76e6f-kube-api-access-s9dnh\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.210101 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.209920 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.210101 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.209941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.210101 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.209992 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-thanos-querier-tls\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.210101 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.210011 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.210101 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.210056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-grpc-tls\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.210347 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.210177 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8c24501-1c77-4804-b703-5ae46ee76e6f-metrics-client-ca\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.311064 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.311025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.311184 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.311071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.311184 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.311108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-thanos-querier-tls\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.311184 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.311136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.311184 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.311171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-grpc-tls\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.311391 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.311211 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8c24501-1c77-4804-b703-5ae46ee76e6f-metrics-client-ca\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.311391 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.311254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.311391 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.311275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9dnh\" (UniqueName: \"kubernetes.io/projected/f8c24501-1c77-4804-b703-5ae46ee76e6f-kube-api-access-s9dnh\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.313021 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.312430 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8c24501-1c77-4804-b703-5ae46ee76e6f-metrics-client-ca\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.315045 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.314688 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-grpc-tls\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.315425 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.315384 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.315906 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.315881 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.316508 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.316483 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-thanos-querier-tls\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.316944 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.316922 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.317310 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.317284 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f8c24501-1c77-4804-b703-5ae46ee76e6f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.319843 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.319822 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9dnh\" (UniqueName: \"kubernetes.io/projected/f8c24501-1c77-4804-b703-5ae46ee76e6f-kube-api-access-s9dnh\") pod \"thanos-querier-6b9f97f8fc-xm6c6\" (UID: \"f8c24501-1c77-4804-b703-5ae46ee76e6f\") " pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.337468 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.337445 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 12 18:18:32.572190 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.572156 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:32.716616 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:32.716583 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6"] Jul 12 18:18:32.719183 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:18:32.719156 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c24501_1c77_4804_b703_5ae46ee76e6f.slice/crio-3b53c4eb2c5b988462abf805ff390d5197ab23281eec314442aa079f0a71c27a WatchSource:0}: Error finding container 3b53c4eb2c5b988462abf805ff390d5197ab23281eec314442aa079f0a71c27a: Status 404 returned error can't find the container with id 3b53c4eb2c5b988462abf805ff390d5197ab23281eec314442aa079f0a71c27a Jul 12 18:18:33.085007 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:33.084925 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:33.145230 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:33.144849 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vpp5s" event={"ID":"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa","Type":"ContainerStarted","Data":"a70c827eabf5d41f38451a62994aa471146db5c6f8f48dee6ab610696291e2da"} Jul 12 18:18:33.145230 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:33.144920 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vpp5s" event={"ID":"9cbfbdb1-45ff-461a-9517-6174e7ce7dfa","Type":"ContainerStarted","Data":"99fa9fede0aab129c92c79d7504a3c80d7155a1f75bd955eb0aff16ac0746e33"} Jul 12 18:18:33.146818 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:33.146715 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" event={"ID":"f8c24501-1c77-4804-b703-5ae46ee76e6f","Type":"ContainerStarted","Data":"3b53c4eb2c5b988462abf805ff390d5197ab23281eec314442aa079f0a71c27a"} Jul 12 18:18:33.149902 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:33.149875 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" event={"ID":"12e428fb-eb05-4442-90c5-183a117de7d5","Type":"ContainerStarted","Data":"25e270430412aa0e0122d4b7293b2866477f9997aa2e250117ae3c713cdd89db"} Jul 12 18:18:33.150025 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:33.149934 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" event={"ID":"12e428fb-eb05-4442-90c5-183a117de7d5","Type":"ContainerStarted","Data":"fa94457efae5bd0647062128bd0599737c9a99b97944fd901d0e5b1e19965de0"} Jul 12 18:18:33.150025 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:33.149956 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" event={"ID":"12e428fb-eb05-4442-90c5-183a117de7d5","Type":"ContainerStarted","Data":"5cc45ae153cbbbb645d9a3cee5372d7c230801fd8a8599fff3d8af1d72031c50"} Jul 12 18:18:33.151137 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:33.151114 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8b090ef-4073-4a60-b30b-c54e39cff6df","Type":"ContainerStarted","Data":"9774c0c70e676be9cb75611e31af693becfcd55957b3d967ff16ed65051bccb8"} Jul 12 18:18:33.161296 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:33.161254 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vpp5s" podStartSLOduration=2.260675979 podStartE2EDuration="3.161241673s" totalImagesPullingTime="829.075023ms" totalInitContainerRuntime="71.490671ms" isStatefulPod=true podCreationTimestamp="2026-07-12 18:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:18:33.161148712 +0000 UTC m=+74.928446297" watchObservedRunningTime="2026-07-12 18:18:33.161241673 +0000 UTC m=+74.928539255" Jul 12 18:18:33.165418 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:33.165393 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:33.623414 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:33.623378 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:34.159965 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:34.159923 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerID="b79b10216d4d1511a6126f1eee1e2ff89f9af3033fb3575a1192bfff1cda0083" exitCode=0 Jul 12 18:18:34.160422 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:34.160086 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8b090ef-4073-4a60-b30b-c54e39cff6df","Type":"ContainerDied","Data":"b79b10216d4d1511a6126f1eee1e2ff89f9af3033fb3575a1192bfff1cda0083"} Jul 12 18:18:34.163079 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:34.163019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" event={"ID":"12e428fb-eb05-4442-90c5-183a117de7d5","Type":"ContainerStarted","Data":"20c9e091fab6aeb5f387ff7f8774ea15fcc0b1dfa93611b95f6e7808dc8ac835"} Jul 12 18:18:34.185130 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:34.185046 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:34.236952 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:34.236881 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xmj8b" podStartSLOduration=2.893819105 podStartE2EDuration="4.23686574s" totalImagesPullingTime="1.343046635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:18:34.235604531 +0000 UTC m=+76.002902115" watchObservedRunningTime="2026-07-12 18:18:34.23686574 +0000 UTC m=+76.004163322" Jul 12 18:18:34.623583 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:34.623496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:34.677206 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:34.677167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:34.939834 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:34.939804 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:34.943510 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:34.943485 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-gvznh"] Jul 12 18:18:34.948339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:34.948315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gvznh" Jul 12 18:18:34.950673 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:34.950631 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 12 18:18:34.950927 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:34.950910 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wkc2d\"" Jul 12 18:18:34.955614 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:34.955546 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-gvznh"] Jul 12 18:18:35.035161 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:35.035132 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4318b9e8-96d1-488f-8df6-8ebd16d3831d-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-gvznh\" (UID: \"4318b9e8-96d1-488f-8df6-8ebd16d3831d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gvznh" Jul 12 18:18:35.035305 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:35.035188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdlcl\" (UniqueName: \"kubernetes.io/projected/4318b9e8-96d1-488f-8df6-8ebd16d3831d-kube-api-access-kdlcl\") pod \"monitoring-plugin-5f68df7f58-gvznh\" (UID: \"4318b9e8-96d1-488f-8df6-8ebd16d3831d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gvznh" Jul 12 18:18:35.037987 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:35.037924 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:35.098129 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:35.097234 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wndcz" Jul 12 18:18:35.119042 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:35.119007 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:35.135946 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:35.135922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4318b9e8-96d1-488f-8df6-8ebd16d3831d-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-gvznh\" (UID: \"4318b9e8-96d1-488f-8df6-8ebd16d3831d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gvznh" Jul 12 18:18:35.136046 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:35.135967 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlcl\" (UniqueName: \"kubernetes.io/projected/4318b9e8-96d1-488f-8df6-8ebd16d3831d-kube-api-access-kdlcl\") pod \"monitoring-plugin-5f68df7f58-gvznh\" (UID: \"4318b9e8-96d1-488f-8df6-8ebd16d3831d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gvznh" Jul 12 18:18:35.146780 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:35.146720 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4318b9e8-96d1-488f-8df6-8ebd16d3831d-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-gvznh\" (UID: \"4318b9e8-96d1-488f-8df6-8ebd16d3831d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gvznh" Jul 12 18:18:35.153720 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:35.153460 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdlcl\" (UniqueName: \"kubernetes.io/projected/4318b9e8-96d1-488f-8df6-8ebd16d3831d-kube-api-access-kdlcl\") pod \"monitoring-plugin-5f68df7f58-gvznh\" (UID: \"4318b9e8-96d1-488f-8df6-8ebd16d3831d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gvznh" Jul 12 18:18:35.175130 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:35.175092 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" event={"ID":"f8c24501-1c77-4804-b703-5ae46ee76e6f","Type":"ContainerStarted","Data":"19fcae0bb07ced8abb4aa73ebbb9c1dc2f7ac7a9669e44d21bc97227c7174cae"} Jul 12 18:18:35.175584 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:35.175183 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" event={"ID":"f8c24501-1c77-4804-b703-5ae46ee76e6f","Type":"ContainerStarted","Data":"d066e27fd925ea803ffdd4b8da68db6a06e522dae5de7f09b3773e1d68e0a120"} Jul 12 18:18:35.175584 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:35.175230 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" event={"ID":"f8c24501-1c77-4804-b703-5ae46ee76e6f","Type":"ContainerStarted","Data":"2cd1d1ee3c34be99ae553fc69a5da9a6121229566bcbcae345984065c255012a"} Jul 12 18:18:35.262691 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:35.262657 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gvznh" Jul 12 18:18:35.415143 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:35.414988 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-gvznh"] Jul 12 18:18:35.417897 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:18:35.417870 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4318b9e8_96d1_488f_8df6_8ebd16d3831d.slice/crio-91de1e8b83b28a2be04141869008d3216b5179a6c09b104d52d196cb9e9afca7 WatchSource:0}: Error finding container 91de1e8b83b28a2be04141869008d3216b5179a6c09b104d52d196cb9e9afca7: Status 404 returned error can't find the container with id 91de1e8b83b28a2be04141869008d3216b5179a6c09b104d52d196cb9e9afca7 Jul 12 18:18:36.183087 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.183010 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gvznh" event={"ID":"4318b9e8-96d1-488f-8df6-8ebd16d3831d","Type":"ContainerStarted","Data":"91de1e8b83b28a2be04141869008d3216b5179a6c09b104d52d196cb9e9afca7"} Jul 12 18:18:36.406813 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.404715 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 12 18:18:36.407560 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.407352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:36.412430 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.410432 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.415320 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.413527 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 12 18:18:36.415320 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.413977 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 12 18:18:36.415320 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.415163 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 12 18:18:36.416990 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.415772 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 12 18:18:36.416990 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.416001 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 12 18:18:36.416990 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.416170 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-65q4l\"" Jul 12 18:18:36.416990 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.416327 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 12 18:18:36.416990 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.416511 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 12 18:18:36.416990 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.416806 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 12 18:18:36.416990 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.416903 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 12 18:18:36.417393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.417119 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 12 18:18:36.419122 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.417706 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 12 18:18:36.419122 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.417925 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 12 18:18:36.419122 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.418915 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 12 18:18:36.419363 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.419276 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4hkereo3jed7c\"" Jul 12 18:18:36.420637 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.420616 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 12 18:18:36.422054 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.422035 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 12 18:18:36.426842 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.424406 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 12 18:18:36.552278 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.551539 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.552278 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.551580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.552278 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.551617 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.552278 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.551644 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.552278 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.551671 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.552278 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.551694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-web-config\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.552278 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.551722 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.552278 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.551750 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88933f4a-7ffc-4874-99da-bbffd9e52b92-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.552278 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.551778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.552278 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.551831 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-config\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.552278 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.551859 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.552278 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.551881 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.552278 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.551903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.552278 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.551923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.552278 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.551950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88933f4a-7ffc-4874-99da-bbffd9e52b92-config-out\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.552278 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.551974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9dx\" (UniqueName: \"kubernetes.io/projected/88933f4a-7ffc-4874-99da-bbffd9e52b92-kube-api-access-6v9dx\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.553053 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.552000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.553053 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.552037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.553053 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.552058 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.553053 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.552083 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.654687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.654687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653077 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.654687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.654687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-web-config\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.654687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.654687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653214 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88933f4a-7ffc-4874-99da-bbffd9e52b92-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.654687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.654687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-config\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.654687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.654687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653354 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.654687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653397 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.654687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.654687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88933f4a-7ffc-4874-99da-bbffd9e52b92-config-out\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.654687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9dx\" (UniqueName: \"kubernetes.io/projected/88933f4a-7ffc-4874-99da-bbffd9e52b92-kube-api-access-6v9dx\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.654687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653565 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.654687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653606 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.654687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.655508 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.655508 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.655508 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.653747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.656970 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.656564 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.657586 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.657207 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.658307 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.658278 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.658766 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.658742 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.658885 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.658840 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.659711 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.659488 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88933f4a-7ffc-4874-99da-bbffd9e52b92-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.659826 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.659753 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.659826 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.659774 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.660415 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.660268 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.660415 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.660390 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.662164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.662121 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.662473 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.662430 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88933f4a-7ffc-4874-99da-bbffd9e52b92-config-out\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.665239 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.665158 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.665239 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.665197 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-config\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.665962 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.665866 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.666420 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.666296 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-web-config\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.667079 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.666649 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.667079 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.666994 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.667914 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.667578 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.673822 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.670348 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9dx\" (UniqueName: \"kubernetes.io/projected/88933f4a-7ffc-4874-99da-bbffd9e52b92-kube-api-access-6v9dx\") pod \"prometheus-k8s-0\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.744958 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.744931 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:36.941142 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:36.941032 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 12 18:18:37.194740 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:37.194702 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" event={"ID":"f8c24501-1c77-4804-b703-5ae46ee76e6f","Type":"ContainerStarted","Data":"af03df5e8adb2e9c2fc5485f50f3c4f62413301e7211d952b67e1a117908a82e"} Jul 12 18:18:37.195238 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:37.194836 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" event={"ID":"f8c24501-1c77-4804-b703-5ae46ee76e6f","Type":"ContainerStarted","Data":"dad8a970cf096c4c2b809ff15fcca6ab9a0321c877ffa0978c38fc5dfa1f23a5"} Jul 12 18:18:37.195238 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:37.194886 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" event={"ID":"f8c24501-1c77-4804-b703-5ae46ee76e6f","Type":"ContainerStarted","Data":"03447e4da7623b9fbb9af8d4a76c170c9d4f2cb7d2f23470bbb23ab2f1203951"} Jul 12 18:18:37.195238 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:37.194935 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:37.196646 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:37.196618 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88933f4a-7ffc-4874-99da-bbffd9e52b92","Type":"ContainerStarted","Data":"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae"} Jul 12 18:18:37.196806 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:37.196737 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88933f4a-7ffc-4874-99da-bbffd9e52b92","Type":"ContainerStarted","Data":"fc6b324fdd93fca40ea5da565ff7e0e1ffd13c224ab348bb9d1327f992f7f315"} Jul 12 18:18:37.200718 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:37.200601 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8b090ef-4073-4a60-b30b-c54e39cff6df","Type":"ContainerStarted","Data":"a070595f86ee29f6110f70ee3532df90d2cc201d0e3d91b8ea1fc4d402f9209f"} Jul 12 18:18:37.200718 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:37.200649 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8b090ef-4073-4a60-b30b-c54e39cff6df","Type":"ContainerStarted","Data":"cef6ba4240ffb0d4903bd0be07be00fbec9c1cabd9e9eb655100a68bb45bec9a"} Jul 12 18:18:37.200718 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:37.200676 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8b090ef-4073-4a60-b30b-c54e39cff6df","Type":"ContainerStarted","Data":"1df9c215fa731bb311897af28bdb69dff93f4e8ff4fdcdd7f3fb2f171bd4b793"} Jul 12 18:18:37.200718 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:37.200705 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8b090ef-4073-4a60-b30b-c54e39cff6df","Type":"ContainerStarted","Data":"4ec6a4317d18c2569cc82251f21bd6d36ead2c9e13486148c6d5e093fd2b6436"} Jul 12 18:18:37.200992 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:37.200742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8b090ef-4073-4a60-b30b-c54e39cff6df","Type":"ContainerStarted","Data":"f29528aa67477609cb9ae901fca3c13a9ff0dbb9416c29bcc6b4439025d72579"} Jul 12 18:18:37.200992 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:37.200805 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8b090ef-4073-4a60-b30b-c54e39cff6df","Type":"ContainerStarted","Data":"1ab8192a76a0f174295ab8a291c1afbb15a6ef51910d698ac189450d7549b043"} Jul 12 18:18:37.214590 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:37.214541 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" podStartSLOduration=1.8010517419999998 podStartE2EDuration="5.214523316s" totalImagesPullingTime="3.413471574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:18:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:18:37.213495674 +0000 UTC m=+78.980793256" watchObservedRunningTime="2026-07-12 18:18:37.214523316 +0000 UTC m=+78.981820899" Jul 12 18:18:37.262459 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:37.261981 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.6312814380000003 podStartE2EDuration="6.261959054s" totalImagesPullingTime="3.607878105s" totalInitContainerRuntime="22.799511ms" isStatefulPod=false podCreationTimestamp="2026-07-12 18:18:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:18:37.260588535 +0000 UTC m=+79.027886118" watchObservedRunningTime="2026-07-12 18:18:37.261959054 +0000 UTC m=+79.029256636" Jul 12 18:18:38.207540 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:38.207482 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gvznh" event={"ID":"4318b9e8-96d1-488f-8df6-8ebd16d3831d","Type":"ContainerStarted","Data":"76a60d0707338ef3e3af699db5e6438ffccc9c3e8baa9fe0c52b75dc3642f55b"} Jul 12 18:18:38.208027 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:38.207650 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gvznh" Jul 12 18:18:38.210653 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:38.210629 2576 generic.go:356] "Generic (PLEG): container finished" podID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerID="519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae" exitCode=0 Jul 12 18:18:38.210840 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:38.210749 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88933f4a-7ffc-4874-99da-bbffd9e52b92","Type":"ContainerDied","Data":"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae"} Jul 12 18:18:38.214815 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:38.214778 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gvznh" Jul 12 18:18:38.224180 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:38.224131 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gvznh" podStartSLOduration=2.08904702 podStartE2EDuration="4.224116574s" totalImagesPullingTime="2.135069554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:18:38.222477863 +0000 UTC m=+79.989775442" watchObservedRunningTime="2026-07-12 18:18:38.224116574 +0000 UTC m=+79.991414158" Jul 12 18:18:39.130902 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:39.130809 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:39.231857 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:39.231821 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:40.058390 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:40.058362 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f788c444b-9sjcx" Jul 12 18:18:40.177847 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:40.177707 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:40.280545 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:40.280492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:40.380698 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:40.380612 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:40.637153 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:40.637056 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:40.741635 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:40.741551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:42.029335 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:42.029289 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:42.130409 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:42.130375 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:42.230376 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:42.230307 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:42.236492 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:42.236464 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88933f4a-7ffc-4874-99da-bbffd9e52b92","Type":"ContainerStarted","Data":"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2"} Jul 12 18:18:42.236586 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:42.236549 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88933f4a-7ffc-4874-99da-bbffd9e52b92","Type":"ContainerStarted","Data":"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289"} Jul 12 18:18:42.236639 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:42.236590 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88933f4a-7ffc-4874-99da-bbffd9e52b92","Type":"ContainerStarted","Data":"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86"} Jul 12 18:18:43.219471 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:43.219433 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6b9f97f8fc-xm6c6" Jul 12 18:18:43.248698 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:43.248662 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88933f4a-7ffc-4874-99da-bbffd9e52b92","Type":"ContainerStarted","Data":"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196"} Jul 12 18:18:43.248877 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:43.248747 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88933f4a-7ffc-4874-99da-bbffd9e52b92","Type":"ContainerStarted","Data":"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a"} Jul 12 18:18:43.248877 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:43.248810 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88933f4a-7ffc-4874-99da-bbffd9e52b92","Type":"ContainerStarted","Data":"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c"} Jul 12 18:18:43.278167 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:43.278103 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.530529934 podStartE2EDuration="7.278084715s" totalImagesPullingTime="3.717462186s" totalInitContainerRuntime="30.092595ms" isStatefulPod=false podCreationTimestamp="2026-07-12 18:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:18:43.276279745 +0000 UTC m=+85.043577332" watchObservedRunningTime="2026-07-12 18:18:43.278084715 +0000 UTC m=+85.045382297" Jul 12 18:18:43.691511 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:43.691480 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:43.791452 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:43.791422 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:45.091650 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:45.091611 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:45.141804 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:45.141690 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:46.746148 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:46.746112 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:18:49.085681 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:49.085627 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:49.186511 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:49.186476 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:49.647325 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:49.647291 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:49.748637 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:49.748598 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:50.887587 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:50.887554 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:51.286102 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:51.286015 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-zldlk" event={"ID":"c6483f3c-7c4a-48d4-90f8-ff4deb9aa0a4","Type":"ContainerStarted","Data":"2800cf9c2586c0b5407fa9b00bf9f2ce5887627166fe7f26df9174c76c3ff114"} Jul 12 18:18:51.286275 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:51.286123 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-zldlk" Jul 12 18:18:51.305440 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:51.305410 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-zldlk" Jul 12 18:18:51.305621 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:51.305561 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-zldlk" podStartSLOduration=2.800600115 podStartE2EDuration="22.305543493s" totalImagesPullingTime="19.504943378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:18:51.302983895 +0000 UTC m=+93.070281468" watchObservedRunningTime="2026-07-12 18:18:51.305543493 +0000 UTC m=+93.072841074" Jul 12 18:18:52.377041 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:52.377002 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:52.435698 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:52.435663 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:54.090696 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:54.090652 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:54.191668 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:54.191619 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:57.690014 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:57.689972 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:57.790769 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:57.790729 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:59.433552 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:59.433454 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:59.534962 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:59.534922 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:18:59.586400 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:18:59.586337 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:19:00.119159 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:00.119124 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:19:00.172292 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:00.172258 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:19:02.140667 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:02.140637 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g7w97" Jul 12 18:19:07.380017 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:07.379989 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:19:07.429568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:07.429536 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:19:10.948697 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:10.948642 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:19:36.746125 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:36.746086 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:36.809713 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:36.809686 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:37.490990 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:37.490957 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:49.370977 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:49.370939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:19:49.483600 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:49.483569 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:19:50.411843 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:50.411807 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 12 18:19:50.412313 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:50.412246 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="alertmanager" containerID="cri-o://1ab8192a76a0f174295ab8a291c1afbb15a6ef51910d698ac189450d7549b043" gracePeriod=120 Jul 12 18:19:50.412376 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:50.412301 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="kube-rbac-proxy-web" containerID="cri-o://4ec6a4317d18c2569cc82251f21bd6d36ead2c9e13486148c6d5e093fd2b6436" gracePeriod=120 Jul 12 18:19:50.412376 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:50.412293 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="kube-rbac-proxy-metric" containerID="cri-o://cef6ba4240ffb0d4903bd0be07be00fbec9c1cabd9e9eb655100a68bb45bec9a" gracePeriod=120 Jul 12 18:19:50.412474 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:50.412361 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="kube-rbac-proxy" containerID="cri-o://1df9c215fa731bb311897af28bdb69dff93f4e8ff4fdcdd7f3fb2f171bd4b793" gracePeriod=120 Jul 12 18:19:50.412474 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:50.412374 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="config-reloader" containerID="cri-o://f29528aa67477609cb9ae901fca3c13a9ff0dbb9416c29bcc6b4439025d72579" gracePeriod=120 Jul 12 18:19:50.412562 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:50.412450 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="prom-label-proxy" containerID="cri-o://a070595f86ee29f6110f70ee3532df90d2cc201d0e3d91b8ea1fc4d402f9209f" gracePeriod=120 Jul 12 18:19:50.534694 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:50.534666 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerID="a070595f86ee29f6110f70ee3532df90d2cc201d0e3d91b8ea1fc4d402f9209f" exitCode=0 Jul 12 18:19:50.534694 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:50.534690 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerID="cef6ba4240ffb0d4903bd0be07be00fbec9c1cabd9e9eb655100a68bb45bec9a" exitCode=0 Jul 12 18:19:50.534694 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:50.534698 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerID="1df9c215fa731bb311897af28bdb69dff93f4e8ff4fdcdd7f3fb2f171bd4b793" exitCode=0 Jul 12 18:19:50.534883 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:50.534703 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerID="f29528aa67477609cb9ae901fca3c13a9ff0dbb9416c29bcc6b4439025d72579" exitCode=0 Jul 12 18:19:50.534883 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:50.534710 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerID="1ab8192a76a0f174295ab8a291c1afbb15a6ef51910d698ac189450d7549b043" exitCode=0 Jul 12 18:19:50.534883 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:50.534735 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8b090ef-4073-4a60-b30b-c54e39cff6df","Type":"ContainerDied","Data":"a070595f86ee29f6110f70ee3532df90d2cc201d0e3d91b8ea1fc4d402f9209f"} Jul 12 18:19:50.534883 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:50.534818 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8b090ef-4073-4a60-b30b-c54e39cff6df","Type":"ContainerDied","Data":"cef6ba4240ffb0d4903bd0be07be00fbec9c1cabd9e9eb655100a68bb45bec9a"} Jul 12 18:19:50.534883 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:50.534848 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8b090ef-4073-4a60-b30b-c54e39cff6df","Type":"ContainerDied","Data":"1df9c215fa731bb311897af28bdb69dff93f4e8ff4fdcdd7f3fb2f171bd4b793"} Jul 12 18:19:50.534883 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:50.534872 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8b090ef-4073-4a60-b30b-c54e39cff6df","Type":"ContainerDied","Data":"f29528aa67477609cb9ae901fca3c13a9ff0dbb9416c29bcc6b4439025d72579"} Jul 12 18:19:50.535053 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:50.534896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8b090ef-4073-4a60-b30b-c54e39cff6df","Type":"ContainerDied","Data":"1ab8192a76a0f174295ab8a291c1afbb15a6ef51910d698ac189450d7549b043"} Jul 12 18:19:51.542982 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.542952 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerID="4ec6a4317d18c2569cc82251f21bd6d36ead2c9e13486148c6d5e093fd2b6436" exitCode=0 Jul 12 18:19:51.543421 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.543019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8b090ef-4073-4a60-b30b-c54e39cff6df","Type":"ContainerDied","Data":"4ec6a4317d18c2569cc82251f21bd6d36ead2c9e13486148c6d5e093fd2b6436"} Jul 12 18:19:51.656410 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.656380 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:19:51.815222 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.815131 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b090ef-4073-4a60-b30b-c54e39cff6df-metrics-client-ca\") pod \"b8b090ef-4073-4a60-b30b-c54e39cff6df\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " Jul 12 18:19:51.815222 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.815187 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-main-tls\") pod \"b8b090ef-4073-4a60-b30b-c54e39cff6df\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " Jul 12 18:19:51.815222 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.815204 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kghhb\" (UniqueName: \"kubernetes.io/projected/b8b090ef-4073-4a60-b30b-c54e39cff6df-kube-api-access-kghhb\") pod \"b8b090ef-4073-4a60-b30b-c54e39cff6df\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " Jul 12 18:19:51.815503 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.815229 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy\") pod \"b8b090ef-4073-4a60-b30b-c54e39cff6df\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " Jul 12 18:19:51.815503 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.815277 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8b090ef-4073-4a60-b30b-c54e39cff6df-tls-assets\") pod \"b8b090ef-4073-4a60-b30b-c54e39cff6df\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " Jul 12 18:19:51.815503 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.815318 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy-web\") pod \"b8b090ef-4073-4a60-b30b-c54e39cff6df\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " Jul 12 18:19:51.815503 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.815349 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b8b090ef-4073-4a60-b30b-c54e39cff6df\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " Jul 12 18:19:51.815503 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.815385 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b090ef-4073-4a60-b30b-c54e39cff6df-alertmanager-trusted-ca-bundle\") pod \"b8b090ef-4073-4a60-b30b-c54e39cff6df\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " Jul 12 18:19:51.815503 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.815430 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b8b090ef-4073-4a60-b30b-c54e39cff6df-alertmanager-main-db\") pod \"b8b090ef-4073-4a60-b30b-c54e39cff6df\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " Jul 12 18:19:51.815503 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.815462 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8b090ef-4073-4a60-b30b-c54e39cff6df-config-out\") pod \"b8b090ef-4073-4a60-b30b-c54e39cff6df\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " Jul 12 18:19:51.815503 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.815500 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-cluster-tls-config\") pod \"b8b090ef-4073-4a60-b30b-c54e39cff6df\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " Jul 12 18:19:51.815913 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.815539 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-config-volume\") pod \"b8b090ef-4073-4a60-b30b-c54e39cff6df\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " Jul 12 18:19:51.815913 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.815571 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-web-config\") pod \"b8b090ef-4073-4a60-b30b-c54e39cff6df\" (UID: \"b8b090ef-4073-4a60-b30b-c54e39cff6df\") " Jul 12 18:19:51.815913 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.815586 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b090ef-4073-4a60-b30b-c54e39cff6df-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "b8b090ef-4073-4a60-b30b-c54e39cff6df" (UID: "b8b090ef-4073-4a60-b30b-c54e39cff6df"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:19:51.816101 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.816072 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b090ef-4073-4a60-b30b-c54e39cff6df-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "b8b090ef-4073-4a60-b30b-c54e39cff6df" (UID: "b8b090ef-4073-4a60-b30b-c54e39cff6df"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:19:51.816346 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.816321 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b090ef-4073-4a60-b30b-c54e39cff6df-metrics-client-ca\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:51.816415 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.816352 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b090ef-4073-4a60-b30b-c54e39cff6df-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:51.816618 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.816594 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b090ef-4073-4a60-b30b-c54e39cff6df-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "b8b090ef-4073-4a60-b30b-c54e39cff6df" (UID: "b8b090ef-4073-4a60-b30b-c54e39cff6df"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:19:51.818211 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.818177 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "b8b090ef-4073-4a60-b30b-c54e39cff6df" (UID: "b8b090ef-4073-4a60-b30b-c54e39cff6df"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:19:51.818566 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.818459 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "b8b090ef-4073-4a60-b30b-c54e39cff6df" (UID: "b8b090ef-4073-4a60-b30b-c54e39cff6df"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:19:51.818566 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.818523 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "b8b090ef-4073-4a60-b30b-c54e39cff6df" (UID: "b8b090ef-4073-4a60-b30b-c54e39cff6df"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:19:51.818566 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.818536 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b090ef-4073-4a60-b30b-c54e39cff6df-config-out" (OuterVolumeSpecName: "config-out") pod "b8b090ef-4073-4a60-b30b-c54e39cff6df" (UID: "b8b090ef-4073-4a60-b30b-c54e39cff6df"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:19:51.818566 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.818543 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "b8b090ef-4073-4a60-b30b-c54e39cff6df" (UID: "b8b090ef-4073-4a60-b30b-c54e39cff6df"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:19:51.818824 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.818608 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b090ef-4073-4a60-b30b-c54e39cff6df-kube-api-access-kghhb" (OuterVolumeSpecName: "kube-api-access-kghhb") pod "b8b090ef-4073-4a60-b30b-c54e39cff6df" (UID: "b8b090ef-4073-4a60-b30b-c54e39cff6df"). InnerVolumeSpecName "kube-api-access-kghhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:19:51.818876 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.818823 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b090ef-4073-4a60-b30b-c54e39cff6df-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b8b090ef-4073-4a60-b30b-c54e39cff6df" (UID: "b8b090ef-4073-4a60-b30b-c54e39cff6df"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:19:51.819682 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.819657 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-config-volume" (OuterVolumeSpecName: "config-volume") pod "b8b090ef-4073-4a60-b30b-c54e39cff6df" (UID: "b8b090ef-4073-4a60-b30b-c54e39cff6df"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:19:51.823487 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.823462 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "b8b090ef-4073-4a60-b30b-c54e39cff6df" (UID: "b8b090ef-4073-4a60-b30b-c54e39cff6df"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:19:51.830008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.829985 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-web-config" (OuterVolumeSpecName: "web-config") pod "b8b090ef-4073-4a60-b30b-c54e39cff6df" (UID: "b8b090ef-4073-4a60-b30b-c54e39cff6df"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:19:51.917461 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.917425 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:51.917461 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.917455 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:51.917461 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.917467 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b8b090ef-4073-4a60-b30b-c54e39cff6df-alertmanager-main-db\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:51.917680 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.917476 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8b090ef-4073-4a60-b30b-c54e39cff6df-config-out\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:51.917680 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.917486 2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-cluster-tls-config\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:51.917680 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.917494 2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-config-volume\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:51.917680 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.917502 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-web-config\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:51.917680 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.917511 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-main-tls\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:51.917680 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.917522 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kghhb\" (UniqueName: \"kubernetes.io/projected/b8b090ef-4073-4a60-b30b-c54e39cff6df-kube-api-access-kghhb\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:51.917680 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.917530 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8b090ef-4073-4a60-b30b-c54e39cff6df-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:51.917680 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:51.917540 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8b090ef-4073-4a60-b30b-c54e39cff6df-tls-assets\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:52.550350 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:52.550317 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8b090ef-4073-4a60-b30b-c54e39cff6df","Type":"ContainerDied","Data":"9774c0c70e676be9cb75611e31af693becfcd55957b3d967ff16ed65051bccb8"} Jul 12 18:19:52.550827 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:52.550385 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 12 18:19:52.550827 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:52.550406 2576 scope.go:117] "RemoveContainer" containerID="a070595f86ee29f6110f70ee3532df90d2cc201d0e3d91b8ea1fc4d402f9209f" Jul 12 18:19:52.558605 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:52.558587 2576 scope.go:117] "RemoveContainer" containerID="cef6ba4240ffb0d4903bd0be07be00fbec9c1cabd9e9eb655100a68bb45bec9a" Jul 12 18:19:52.565695 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:52.565679 2576 scope.go:117] "RemoveContainer" containerID="1df9c215fa731bb311897af28bdb69dff93f4e8ff4fdcdd7f3fb2f171bd4b793" Jul 12 18:19:52.571599 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:52.571579 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 12 18:19:52.573632 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:52.573607 2576 scope.go:117] "RemoveContainer" containerID="4ec6a4317d18c2569cc82251f21bd6d36ead2c9e13486148c6d5e093fd2b6436" Jul 12 18:19:52.575004 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:52.574982 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 12 18:19:52.580515 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:52.580494 2576 scope.go:117] "RemoveContainer" containerID="f29528aa67477609cb9ae901fca3c13a9ff0dbb9416c29bcc6b4439025d72579" Jul 12 18:19:52.587355 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:52.587337 2576 scope.go:117] "RemoveContainer" containerID="1ab8192a76a0f174295ab8a291c1afbb15a6ef51910d698ac189450d7549b043" Jul 12 18:19:52.594435 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:52.594417 2576 scope.go:117] "RemoveContainer" containerID="b79b10216d4d1511a6126f1eee1e2ff89f9af3033fb3575a1192bfff1cda0083" Jul 12 18:19:52.818168 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:52.818089 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" path="/var/lib/kubelet/pods/b8b090ef-4073-4a60-b30b-c54e39cff6df/volumes" Jul 12 18:19:54.729620 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:54.729587 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 12 18:19:54.730610 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:54.730575 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="prometheus" containerID="cri-o://4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86" gracePeriod=600 Jul 12 18:19:54.730880 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:54.730843 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="thanos-sidecar" containerID="cri-o://4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2" gracePeriod=600 Jul 12 18:19:54.730982 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:54.730874 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a" gracePeriod=600 Jul 12 18:19:54.730982 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:54.730806 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="kube-rbac-proxy" containerID="cri-o://10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c" gracePeriod=600 Jul 12 18:19:54.730982 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:54.730939 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="kube-rbac-proxy-web" containerID="cri-o://b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196" gracePeriod=600 Jul 12 18:19:54.731140 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:54.730988 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="config-reloader" containerID="cri-o://9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289" gracePeriod=600 Jul 12 18:19:54.986573 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:54.986508 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.150231 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150199 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-2\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150231 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150237 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150479 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150255 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-kube-rbac-proxy\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150479 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150272 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-config\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150479 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150287 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-metrics-client-ca\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150479 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150307 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-metrics-client-certs\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150479 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150346 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-thanos-prometheus-http-client-file\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150479 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150375 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-kubelet-serving-ca-bundle\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150479 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150403 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-web-config\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150479 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150427 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v9dx\" (UniqueName: \"kubernetes.io/projected/88933f4a-7ffc-4874-99da-bbffd9e52b92-kube-api-access-6v9dx\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150479 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150459 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-tls\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150944 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150490 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88933f4a-7ffc-4874-99da-bbffd9e52b92-config-out\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150944 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150532 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-0\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150944 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150574 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-grpc-tls\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150944 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150603 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-serving-certs-ca-bundle\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150944 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150650 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-trusted-ca-bundle\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150944 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150673 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88933f4a-7ffc-4874-99da-bbffd9e52b92-tls-assets\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150944 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150668 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:19:55.150944 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150729 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150944 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150755 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-1\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150944 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150812 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-db\") pod \"88933f4a-7ffc-4874-99da-bbffd9e52b92\" (UID: \"88933f4a-7ffc-4874-99da-bbffd9e52b92\") " Jul 12 18:19:55.150944 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.150914 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:19:55.151481 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.151057 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.151481 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.151076 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-metrics-client-ca\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.151481 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.151316 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:19:55.152120 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.152089 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:19:55.153283 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.153014 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-config" (OuterVolumeSpecName: "config") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:19:55.153283 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.153088 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:19:55.153283 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.153111 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:19:55.153283 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.153190 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88933f4a-7ffc-4874-99da-bbffd9e52b92-config-out" (OuterVolumeSpecName: "config-out") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:19:55.153553 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.153351 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:19:55.153553 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.153444 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:19:55.153650 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.153624 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:19:55.153852 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.153829 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:19:55.153948 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.153868 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:19:55.154279 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.154252 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88933f4a-7ffc-4874-99da-bbffd9e52b92-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:19:55.154488 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.154466 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:19:55.154667 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.154635 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:19:55.155333 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.155308 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:19:55.155699 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.155679 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88933f4a-7ffc-4874-99da-bbffd9e52b92-kube-api-access-6v9dx" (OuterVolumeSpecName: "kube-api-access-6v9dx") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "kube-api-access-6v9dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:19:55.155774 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.155712 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:19:55.164181 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.164156 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-web-config" (OuterVolumeSpecName: "web-config") pod "88933f4a-7ffc-4874-99da-bbffd9e52b92" (UID: "88933f4a-7ffc-4874-99da-bbffd9e52b92"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:19:55.252194 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252108 2576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-grpc-tls\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.252194 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252139 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.252194 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252149 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.252194 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252158 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88933f4a-7ffc-4874-99da-bbffd9e52b92-tls-assets\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.252194 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252169 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.252194 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252178 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.252194 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252186 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-db\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.252194 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252195 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.252194 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252204 2576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-kube-rbac-proxy\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.252577 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252212 2576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-config\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.252577 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252221 2576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-metrics-client-certs\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.252577 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252229 2576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.252577 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252238 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.252577 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252248 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-web-config\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.252577 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252255 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v9dx\" (UniqueName: \"kubernetes.io/projected/88933f4a-7ffc-4874-99da-bbffd9e52b92-kube-api-access-6v9dx\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.252577 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252263 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/88933f4a-7ffc-4874-99da-bbffd9e52b92-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.252577 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252271 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88933f4a-7ffc-4874-99da-bbffd9e52b92-config-out\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.252577 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.252279 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/88933f4a-7ffc-4874-99da-bbffd9e52b92-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:19:55.566462 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.566371 2576 generic.go:356] "Generic (PLEG): container finished" podID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerID="b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a" exitCode=0 Jul 12 18:19:55.566462 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.566399 2576 generic.go:356] "Generic (PLEG): container finished" podID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerID="10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c" exitCode=0 Jul 12 18:19:55.566462 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.566406 2576 generic.go:356] "Generic (PLEG): container finished" podID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerID="b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196" exitCode=0 Jul 12 18:19:55.566462 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.566412 2576 generic.go:356] "Generic (PLEG): container finished" podID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerID="4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2" exitCode=0 Jul 12 18:19:55.566462 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.566417 2576 generic.go:356] "Generic (PLEG): container finished" podID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerID="9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289" exitCode=0 Jul 12 18:19:55.566462 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.566422 2576 generic.go:356] "Generic (PLEG): container finished" podID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerID="4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86" exitCode=0 Jul 12 18:19:55.566462 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.566455 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88933f4a-7ffc-4874-99da-bbffd9e52b92","Type":"ContainerDied","Data":"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a"} Jul 12 18:19:55.566462 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.566462 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.566960 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.566517 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88933f4a-7ffc-4874-99da-bbffd9e52b92","Type":"ContainerDied","Data":"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c"} Jul 12 18:19:55.566960 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.566542 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88933f4a-7ffc-4874-99da-bbffd9e52b92","Type":"ContainerDied","Data":"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196"} Jul 12 18:19:55.566960 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.566566 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88933f4a-7ffc-4874-99da-bbffd9e52b92","Type":"ContainerDied","Data":"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2"} Jul 12 18:19:55.566960 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.566596 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88933f4a-7ffc-4874-99da-bbffd9e52b92","Type":"ContainerDied","Data":"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289"} Jul 12 18:19:55.566960 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.566595 2576 scope.go:117] "RemoveContainer" containerID="b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a" Jul 12 18:19:55.566960 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.566622 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88933f4a-7ffc-4874-99da-bbffd9e52b92","Type":"ContainerDied","Data":"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86"} Jul 12 18:19:55.566960 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.566653 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88933f4a-7ffc-4874-99da-bbffd9e52b92","Type":"ContainerDied","Data":"fc6b324fdd93fca40ea5da565ff7e0e1ffd13c224ab348bb9d1327f992f7f315"} Jul 12 18:19:55.575166 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.575149 2576 scope.go:117] "RemoveContainer" containerID="10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c" Jul 12 18:19:55.581923 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.581904 2576 scope.go:117] "RemoveContainer" containerID="b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196" Jul 12 18:19:55.591196 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.591173 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 12 18:19:55.592077 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.592025 2576 scope.go:117] "RemoveContainer" containerID="4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2" Jul 12 18:19:55.593406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.593391 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 12 18:19:55.599243 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.599226 2576 scope.go:117] "RemoveContainer" containerID="9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289" Jul 12 18:19:55.606148 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.606130 2576 scope.go:117] "RemoveContainer" containerID="4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86" Jul 12 18:19:55.613137 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.613120 2576 scope.go:117] "RemoveContainer" containerID="519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae" Jul 12 18:19:55.617633 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.617612 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 12 18:19:55.618040 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618023 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="kube-rbac-proxy-metric" Jul 12 18:19:55.618094 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618045 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="kube-rbac-proxy-metric" Jul 12 18:19:55.618094 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618064 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="prom-label-proxy" Jul 12 18:19:55.618094 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618072 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="prom-label-proxy" Jul 12 18:19:55.618094 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618082 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="init-config-reloader" Jul 12 18:19:55.618094 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618092 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="init-config-reloader" Jul 12 18:19:55.618234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618106 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="alertmanager" Jul 12 18:19:55.618234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618116 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="alertmanager" Jul 12 18:19:55.618234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618125 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="kube-rbac-proxy-web" Jul 12 18:19:55.618234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618134 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="kube-rbac-proxy-web" Jul 12 18:19:55.618234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618145 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="config-reloader" Jul 12 18:19:55.618234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618153 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="config-reloader" Jul 12 18:19:55.618234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618171 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="kube-rbac-proxy" Jul 12 18:19:55.618234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618179 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="kube-rbac-proxy" Jul 12 18:19:55.618234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618188 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="thanos-sidecar" Jul 12 18:19:55.618234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618196 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="thanos-sidecar" Jul 12 18:19:55.618234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618208 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="init-config-reloader" Jul 12 18:19:55.618234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618215 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="init-config-reloader" Jul 12 18:19:55.618234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618228 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="config-reloader" Jul 12 18:19:55.618234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618236 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="config-reloader" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618253 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="kube-rbac-proxy-web" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618261 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="kube-rbac-proxy-web" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618271 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="kube-rbac-proxy-thanos" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618279 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="kube-rbac-proxy-thanos" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618290 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="prometheus" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618298 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="prometheus" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618307 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="kube-rbac-proxy" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618316 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="kube-rbac-proxy" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618385 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="kube-rbac-proxy" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618397 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="prom-label-proxy" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618407 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="config-reloader" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618418 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="kube-rbac-proxy-metric" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618426 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="kube-rbac-proxy-thanos" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618434 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="prometheus" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618444 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="kube-rbac-proxy-web" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618452 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="kube-rbac-proxy" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618462 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="config-reloader" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618470 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" containerName="thanos-sidecar" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618478 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="alertmanager" Jul 12 18:19:55.618629 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.618485 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b090ef-4073-4a60-b30b-c54e39cff6df" containerName="kube-rbac-proxy-web" Jul 12 18:19:55.621044 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.621020 2576 scope.go:117] "RemoveContainer" containerID="b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a" Jul 12 18:19:55.621305 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.621290 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.621385 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:19:55.621339 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a\": container with ID starting with b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a not found: ID does not exist" containerID="b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a" Jul 12 18:19:55.621430 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.621366 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a"} err="failed to get container status \"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a\": rpc error: code = NotFound desc = could not find container \"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a\": container with ID starting with b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a not found: ID does not exist" Jul 12 18:19:55.621430 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.621396 2576 scope.go:117] "RemoveContainer" containerID="10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c" Jul 12 18:19:55.621839 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:19:55.621815 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c\": container with ID starting with 10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c not found: ID does not exist" containerID="10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c" Jul 12 18:19:55.621915 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.621845 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c"} err="failed to get container status \"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c\": rpc error: code = NotFound desc = could not find container \"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c\": container with ID starting with 10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c not found: ID does not exist" Jul 12 18:19:55.621915 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.621865 2576 scope.go:117] "RemoveContainer" containerID="b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196" Jul 12 18:19:55.622132 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:19:55.622108 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196\": container with ID starting with b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196 not found: ID does not exist" containerID="b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196" Jul 12 18:19:55.622184 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.622142 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196"} err="failed to get container status \"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196\": rpc error: code = NotFound desc = could not find container \"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196\": container with ID starting with b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196 not found: ID does not exist" Jul 12 18:19:55.622184 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.622165 2576 scope.go:117] "RemoveContainer" containerID="4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2" Jul 12 18:19:55.622403 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:19:55.622387 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2\": container with ID starting with 4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2 not found: ID does not exist" containerID="4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2" Jul 12 18:19:55.622449 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.622407 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2"} err="failed to get container status \"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2\": rpc error: code = NotFound desc = could not find container \"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2\": container with ID starting with 4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2 not found: ID does not exist" Jul 12 18:19:55.622449 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.622423 2576 scope.go:117] "RemoveContainer" containerID="9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289" Jul 12 18:19:55.622682 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:19:55.622668 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289\": container with ID starting with 9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289 not found: ID does not exist" containerID="9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289" Jul 12 18:19:55.622720 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.622686 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289"} err="failed to get container status \"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289\": rpc error: code = NotFound desc = could not find container \"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289\": container with ID starting with 9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289 not found: ID does not exist" Jul 12 18:19:55.622720 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.622698 2576 scope.go:117] "RemoveContainer" containerID="4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86" Jul 12 18:19:55.622994 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:19:55.622975 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86\": container with ID starting with 4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86 not found: ID does not exist" containerID="4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86" Jul 12 18:19:55.623082 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.622997 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86"} err="failed to get container status \"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86\": rpc error: code = NotFound desc = could not find container \"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86\": container with ID starting with 4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86 not found: ID does not exist" Jul 12 18:19:55.623082 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.623011 2576 scope.go:117] "RemoveContainer" containerID="519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae" Jul 12 18:19:55.623264 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:19:55.623248 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae\": container with ID starting with 519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae not found: ID does not exist" containerID="519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae" Jul 12 18:19:55.623322 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.623266 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae"} err="failed to get container status \"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae\": rpc error: code = NotFound desc = could not find container \"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae\": container with ID starting with 519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae not found: ID does not exist" Jul 12 18:19:55.623322 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.623280 2576 scope.go:117] "RemoveContainer" containerID="b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a" Jul 12 18:19:55.623534 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.623514 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a"} err="failed to get container status \"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a\": rpc error: code = NotFound desc = could not find container \"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a\": container with ID starting with b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a not found: ID does not exist" Jul 12 18:19:55.623534 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.623532 2576 scope.go:117] "RemoveContainer" containerID="10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c" Jul 12 18:19:55.623844 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.623820 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c"} err="failed to get container status \"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c\": rpc error: code = NotFound desc = could not find container \"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c\": container with ID starting with 10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c not found: ID does not exist" Jul 12 18:19:55.623844 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.623841 2576 scope.go:117] "RemoveContainer" containerID="b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196" Jul 12 18:19:55.624164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.624138 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 12 18:19:55.624267 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.624245 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 12 18:19:55.624334 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.624144 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4hkereo3jed7c\"" Jul 12 18:19:55.624385 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.624331 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196"} err="failed to get container status \"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196\": rpc error: code = NotFound desc = could not find container \"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196\": container with ID starting with b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196 not found: ID does not exist" Jul 12 18:19:55.624385 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.624376 2576 scope.go:117] "RemoveContainer" containerID="4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2" Jul 12 18:19:55.624580 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.624563 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 12 18:19:55.624658 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.624639 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 12 18:19:55.624899 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.624879 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 12 18:19:55.625002 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.624890 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 12 18:19:55.625111 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.624996 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2"} err="failed to get container status \"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2\": rpc error: code = NotFound desc = could not find container \"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2\": container with ID starting with 4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2 not found: ID does not exist" Jul 12 18:19:55.625180 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.625111 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 12 18:19:55.625253 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.625076 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 12 18:19:55.625302 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.625116 2576 scope.go:117] "RemoveContainer" containerID="9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289" Jul 12 18:19:55.625355 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.625027 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 12 18:19:55.625957 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.625535 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 12 18:19:55.625957 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.625700 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 12 18:19:55.625957 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.625807 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 12 18:19:55.626219 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.625952 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289"} err="failed to get container status \"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289\": rpc error: code = NotFound desc = could not find container \"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289\": container with ID starting with 9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289 not found: ID does not exist" Jul 12 18:19:55.626219 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.625972 2576 scope.go:117] "RemoveContainer" containerID="4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86" Jul 12 18:19:55.626354 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.626330 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86"} err="failed to get container status \"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86\": rpc error: code = NotFound desc = could not find container \"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86\": container with ID starting with 4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86 not found: ID does not exist" Jul 12 18:19:55.626447 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.626354 2576 scope.go:117] "RemoveContainer" containerID="519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae" Jul 12 18:19:55.626666 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.626627 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae"} err="failed to get container status \"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae\": rpc error: code = NotFound desc = could not find container \"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae\": container with ID starting with 519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae not found: ID does not exist" Jul 12 18:19:55.626666 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.626652 2576 scope.go:117] "RemoveContainer" containerID="b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a" Jul 12 18:19:55.627019 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.627003 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 12 18:19:55.627340 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.627303 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a"} err="failed to get container status \"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a\": rpc error: code = NotFound desc = could not find container \"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a\": container with ID starting with b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a not found: ID does not exist" Jul 12 18:19:55.627340 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.627332 2576 scope.go:117] "RemoveContainer" containerID="10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c" Jul 12 18:19:55.627898 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.627551 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-65q4l\"" Jul 12 18:19:55.627997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.627906 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 12 18:19:55.628254 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.628199 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c"} err="failed to get container status \"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c\": rpc error: code = NotFound desc = could not find container \"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c\": container with ID starting with 10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c not found: ID does not exist" Jul 12 18:19:55.628254 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.628228 2576 scope.go:117] "RemoveContainer" containerID="b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196" Jul 12 18:19:55.628663 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.628643 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196"} err="failed to get container status \"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196\": rpc error: code = NotFound desc = could not find container \"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196\": container with ID starting with b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196 not found: ID does not exist" Jul 12 18:19:55.628758 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.628665 2576 scope.go:117] "RemoveContainer" containerID="4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2" Jul 12 18:19:55.628991 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.628967 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2"} err="failed to get container status \"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2\": rpc error: code = NotFound desc = could not find container \"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2\": container with ID starting with 4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2 not found: ID does not exist" Jul 12 18:19:55.628991 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.628991 2576 scope.go:117] "RemoveContainer" containerID="9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289" Jul 12 18:19:55.629328 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.629297 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289"} err="failed to get container status \"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289\": rpc error: code = NotFound desc = could not find container \"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289\": container with ID starting with 9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289 not found: ID does not exist" Jul 12 18:19:55.629328 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.629325 2576 scope.go:117] "RemoveContainer" containerID="4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86" Jul 12 18:19:55.629704 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.629593 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86"} err="failed to get container status \"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86\": rpc error: code = NotFound desc = could not find container \"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86\": container with ID starting with 4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86 not found: ID does not exist" Jul 12 18:19:55.629704 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.629617 2576 scope.go:117] "RemoveContainer" containerID="519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae" Jul 12 18:19:55.630180 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.630138 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae"} err="failed to get container status \"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae\": rpc error: code = NotFound desc = could not find container \"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae\": container with ID starting with 519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae not found: ID does not exist" Jul 12 18:19:55.630180 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.630169 2576 scope.go:117] "RemoveContainer" containerID="b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a" Jul 12 18:19:55.630477 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.630452 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a"} err="failed to get container status \"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a\": rpc error: code = NotFound desc = could not find container \"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a\": container with ID starting with b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a not found: ID does not exist" Jul 12 18:19:55.630551 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.630481 2576 scope.go:117] "RemoveContainer" containerID="10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c" Jul 12 18:19:55.630551 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.630490 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 12 18:19:55.630815 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.630778 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c"} err="failed to get container status \"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c\": rpc error: code = NotFound desc = could not find container \"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c\": container with ID starting with 10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c not found: ID does not exist" Jul 12 18:19:55.630896 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.630817 2576 scope.go:117] "RemoveContainer" containerID="b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196" Jul 12 18:19:55.631047 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.631028 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196"} err="failed to get container status \"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196\": rpc error: code = NotFound desc = could not find container \"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196\": container with ID starting with b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196 not found: ID does not exist" Jul 12 18:19:55.631101 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.631049 2576 scope.go:117] "RemoveContainer" containerID="4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2" Jul 12 18:19:55.631275 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.631256 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2"} err="failed to get container status \"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2\": rpc error: code = NotFound desc = could not find container \"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2\": container with ID starting with 4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2 not found: ID does not exist" Jul 12 18:19:55.631346 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.631276 2576 scope.go:117] "RemoveContainer" containerID="9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289" Jul 12 18:19:55.631504 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.631487 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289"} err="failed to get container status \"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289\": rpc error: code = NotFound desc = could not find container \"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289\": container with ID starting with 9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289 not found: ID does not exist" Jul 12 18:19:55.631569 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.631505 2576 scope.go:117] "RemoveContainer" containerID="4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86" Jul 12 18:19:55.631723 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.631706 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86"} err="failed to get container status \"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86\": rpc error: code = NotFound desc = could not find container \"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86\": container with ID starting with 4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86 not found: ID does not exist" Jul 12 18:19:55.631824 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.631725 2576 scope.go:117] "RemoveContainer" containerID="519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae" Jul 12 18:19:55.636306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.632507 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae"} err="failed to get container status \"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae\": rpc error: code = NotFound desc = could not find container \"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae\": container with ID starting with 519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae not found: ID does not exist" Jul 12 18:19:55.636306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.632540 2576 scope.go:117] "RemoveContainer" containerID="b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a" Jul 12 18:19:55.636843 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.636808 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a"} err="failed to get container status \"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a\": rpc error: code = NotFound desc = could not find container \"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a\": container with ID starting with b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a not found: ID does not exist" Jul 12 18:19:55.636843 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.636842 2576 scope.go:117] "RemoveContainer" containerID="10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c" Jul 12 18:19:55.637116 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.637099 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c"} err="failed to get container status \"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c\": rpc error: code = NotFound desc = could not find container \"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c\": container with ID starting with 10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c not found: ID does not exist" Jul 12 18:19:55.637179 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.637119 2576 scope.go:117] "RemoveContainer" containerID="b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196" Jul 12 18:19:55.637338 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.637320 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196"} err="failed to get container status \"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196\": rpc error: code = NotFound desc = could not find container \"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196\": container with ID starting with b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196 not found: ID does not exist" Jul 12 18:19:55.637445 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.637339 2576 scope.go:117] "RemoveContainer" containerID="4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2" Jul 12 18:19:55.637587 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.637565 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2"} err="failed to get container status \"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2\": rpc error: code = NotFound desc = could not find container \"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2\": container with ID starting with 4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2 not found: ID does not exist" Jul 12 18:19:55.637633 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.637590 2576 scope.go:117] "RemoveContainer" containerID="9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289" Jul 12 18:19:55.637871 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.637849 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289"} err="failed to get container status \"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289\": rpc error: code = NotFound desc = could not find container \"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289\": container with ID starting with 9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289 not found: ID does not exist" Jul 12 18:19:55.637871 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.637869 2576 scope.go:117] "RemoveContainer" containerID="4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86" Jul 12 18:19:55.638138 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.638118 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86"} err="failed to get container status \"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86\": rpc error: code = NotFound desc = could not find container \"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86\": container with ID starting with 4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86 not found: ID does not exist" Jul 12 18:19:55.638288 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.638149 2576 scope.go:117] "RemoveContainer" containerID="519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae" Jul 12 18:19:55.638635 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.638526 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae"} err="failed to get container status \"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae\": rpc error: code = NotFound desc = could not find container \"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae\": container with ID starting with 519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae not found: ID does not exist" Jul 12 18:19:55.638635 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.638548 2576 scope.go:117] "RemoveContainer" containerID="b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a" Jul 12 18:19:55.638856 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.638817 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a"} err="failed to get container status \"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a\": rpc error: code = NotFound desc = could not find container \"b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a\": container with ID starting with b5daef2537031cdcee93910ab65cfe65ec14f4d7805ce6b07a864a5bea63010a not found: ID does not exist" Jul 12 18:19:55.638856 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.638842 2576 scope.go:117] "RemoveContainer" containerID="10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c" Jul 12 18:19:55.639211 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.639174 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c"} err="failed to get container status \"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c\": rpc error: code = NotFound desc = could not find container \"10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c\": container with ID starting with 10a50efc398c1614635e494ef4466a6db00e77f28c586f6aebe969420ae7543c not found: ID does not exist" Jul 12 18:19:55.639412 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.639329 2576 scope.go:117] "RemoveContainer" containerID="b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196" Jul 12 18:19:55.640726 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.640687 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196"} err="failed to get container status \"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196\": rpc error: code = NotFound desc = could not find container \"b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196\": container with ID starting with b0d703c5d1085b93584d7848dc94531dc032dc051d83a9ffd7eb9484c6d49196 not found: ID does not exist" Jul 12 18:19:55.640855 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.640732 2576 scope.go:117] "RemoveContainer" containerID="4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2" Jul 12 18:19:55.641198 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.641166 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2"} err="failed to get container status \"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2\": rpc error: code = NotFound desc = could not find container \"4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2\": container with ID starting with 4bceef25a04d1c5e1d16bdc5d31e2b0a9bf5bdb98373077651976e2244058bd2 not found: ID does not exist" Jul 12 18:19:55.641273 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.641205 2576 scope.go:117] "RemoveContainer" containerID="9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289" Jul 12 18:19:55.641521 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.641498 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289"} err="failed to get container status \"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289\": rpc error: code = NotFound desc = could not find container \"9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289\": container with ID starting with 9660f15121976a40ba21db1bec4dc5ecba8d75b713cb7814dc82ef1647c5f289 not found: ID does not exist" Jul 12 18:19:55.641582 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.641528 2576 scope.go:117] "RemoveContainer" containerID="4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86" Jul 12 18:19:55.641620 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.641509 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 12 18:19:55.641769 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.641751 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86"} err="failed to get container status \"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86\": rpc error: code = NotFound desc = could not find container \"4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86\": container with ID starting with 4820e1447f3199d9c5ef2a463952f8b21ec500be298baf1cdc068c88c5ca2d86 not found: ID does not exist" Jul 12 18:19:55.641844 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.641770 2576 scope.go:117] "RemoveContainer" containerID="519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae" Jul 12 18:19:55.641990 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.641974 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae"} err="failed to get container status \"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae\": rpc error: code = NotFound desc = could not find container \"519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae\": container with ID starting with 519ee48b8b40c2f85caf8feef7dea00af82f376d0a05b9fd44b9bc6114f0b7ae not found: ID does not exist" Jul 12 18:19:55.655279 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655252 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655280 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-web-config\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655300 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655375 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655361 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655499 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655382 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655499 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655499 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655591 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655591 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655591 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655532 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655591 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655550 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655704 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655594 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655704 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655704 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655704 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655682 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-config-out\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655841 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655709 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-config\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655841 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655727 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655841 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655742 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.655841 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.655766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcqq2\" (UniqueName: \"kubernetes.io/projected/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-kube-api-access-zcqq2\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.756445 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.756445 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756450 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.756920 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.756920 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.756920 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.756920 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756574 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.756920 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.756920 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756620 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-config-out\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.756920 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756645 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-config\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.756920 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.756920 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.756920 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcqq2\" (UniqueName: \"kubernetes.io/projected/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-kube-api-access-zcqq2\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.756920 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.756920 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-web-config\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.756920 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.756920 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756905 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.757596 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.757596 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.756981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.757596 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.757024 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.757596 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.757052 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.757596 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.757367 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.757596 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.757436 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.757596 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.757479 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.759821 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.759617 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.759821 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.759617 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-config\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.759821 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.759800 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-web-config\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.759821 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.759810 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.761139 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.759959 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.761139 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.760225 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.761139 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.760271 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.761139 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.760273 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.761139 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.760759 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.761139 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.760883 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.761431 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.761236 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.761711 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.761679 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-config-out\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.762424 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.762396 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.762424 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.762400 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.762564 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.762497 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.763229 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.763211 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.767633 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.767613 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcqq2\" (UniqueName: \"kubernetes.io/projected/fea1ac3c-e030-4b05-a1aa-3b104ee07dc5-kube-api-access-zcqq2\") pod \"prometheus-k8s-0\" (UID: \"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:55.933526 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:55.933479 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:19:56.068478 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:56.068453 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 12 18:19:56.070701 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:19:56.070670 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea1ac3c_e030_4b05_a1aa_3b104ee07dc5.slice/crio-e2921bfdc0673368ccee7421b7395547b41cefc48b3928d699dc0dc38e557aa3 WatchSource:0}: Error finding container e2921bfdc0673368ccee7421b7395547b41cefc48b3928d699dc0dc38e557aa3: Status 404 returned error can't find the container with id e2921bfdc0673368ccee7421b7395547b41cefc48b3928d699dc0dc38e557aa3 Jul 12 18:19:56.571897 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:56.571862 2576 generic.go:356] "Generic (PLEG): container finished" podID="fea1ac3c-e030-4b05-a1aa-3b104ee07dc5" containerID="084712a67f82b5e9f34b98485c7424d826c49fa0b6a4ab2b9d032c36a8d26323" exitCode=0 Jul 12 18:19:56.572073 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:56.571968 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5","Type":"ContainerDied","Data":"084712a67f82b5e9f34b98485c7424d826c49fa0b6a4ab2b9d032c36a8d26323"} Jul 12 18:19:56.572073 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:56.572047 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5","Type":"ContainerStarted","Data":"e2921bfdc0673368ccee7421b7395547b41cefc48b3928d699dc0dc38e557aa3"} Jul 12 18:19:56.817806 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:56.817760 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88933f4a-7ffc-4874-99da-bbffd9e52b92" path="/var/lib/kubelet/pods/88933f4a-7ffc-4874-99da-bbffd9e52b92/volumes" Jul 12 18:19:57.580178 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:57.580143 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5","Type":"ContainerStarted","Data":"798a783a83490f735967f79818f1e1d64ab8f3643240bd6225a0cfcb00402505"} Jul 12 18:19:57.580346 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:57.580223 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5","Type":"ContainerStarted","Data":"d2411597055dab51842aed8082adde3b563a4fe541a0e486298ce0512d30f7f4"} Jul 12 18:19:57.580346 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:57.580271 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5","Type":"ContainerStarted","Data":"2204a84b30a720ca81a1fd214a6cf65b1ac4dc8076771d0bcf661ba3a607a381"} Jul 12 18:19:57.580346 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:57.580305 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5","Type":"ContainerStarted","Data":"eceda0b05af5f6c5a7b164b250e12029b8a81c506132e134394db8697d2371bb"} Jul 12 18:19:57.580346 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:57.580340 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5","Type":"ContainerStarted","Data":"61726a83941b9e609170a1caf37131cc72a7b8227aa810239734aafc814a1193"} Jul 12 18:19:57.580496 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:57.580379 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fea1ac3c-e030-4b05-a1aa-3b104ee07dc5","Type":"ContainerStarted","Data":"765bfa67bbc287f125a22ad4a50d6fd1b93e25662f7ed49d8d852e5e1f0e60f2"} Jul 12 18:19:57.607634 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:57.607573 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.588463194 podStartE2EDuration="2.607552443s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.089249ms" isStatefulPod=false podCreationTimestamp="2026-07-12 18:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:19:57.606250436 +0000 UTC m=+159.373548052" watchObservedRunningTime="2026-07-12 18:19:57.607552443 +0000 UTC m=+159.374850026" Jul 12 18:19:58.530584 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:58.528976 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:19:58.583905 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:19:58.583838 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:20:00.934164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:20:00.934126 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:20:09.080625 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:20:09.080587 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:20:09.132548 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:20:09.132512 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:20:09.184687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:20:09.184656 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:20:38.021680 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:20:38.021643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:20:38.075642 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:20:38.075612 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:20:55.933879 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:20:55.933825 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:20:55.950234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:20:55.950199 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:20:56.808167 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:20:56.808138 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 18:21:04.639094 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:04.639053 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:21:04.694504 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:04.694473 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:21:07.606421 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:07.606389 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:21:07.662318 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:07.662269 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:21:07.748505 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:07.748477 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:21:07.806360 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:07.806319 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:21:08.350539 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:08.350510 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:21:08.404956 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:08.404922 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:21:16.494834 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:16.494779 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:21:16.596186 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:16.596152 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:21:18.325056 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:18.325021 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:21:18.377189 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:18.377145 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:21:18.432000 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:18.431962 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:21:22.387013 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:22.386973 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:21:22.486958 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:22.486905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:21:22.768314 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:22.768217 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:21:51.357704 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:51.357675 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:21:52.566021 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:21:52.565987 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:17.931314 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:17.931272 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:18.643755 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:18.643728 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 12 18:22:18.703458 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:18.703428 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:21.576627 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:21.576576 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:21.627640 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:21.627596 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:22.841110 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:22.841075 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:22.938126 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:22.938081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:23.523651 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:23.523596 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:23.622527 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:23.622495 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:23.697911 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:23.697880 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:25.531228 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:25.531195 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:25.924015 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:25.923984 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:28.715978 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:28.715942 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:28.764926 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:28.764873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:34.125294 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:34.125256 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:36.200140 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:36.200107 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:36.296084 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:36.296044 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:41.188898 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:41.188865 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:41.285389 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:41.285358 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:44.062922 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:44.062872 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:59.866138 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:59.866067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:22:59.965432 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:22:59.965396 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:00.374507 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.374477 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:00.546204 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.546174 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:00.645808 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.645687 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:00.680128 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.680096 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd"] Jul 12 18:23:00.682397 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.682381 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:00.685072 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.685047 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 12 18:23:00.685072 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.685065 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 12 18:23:00.685304 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.685289 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 12 18:23:00.685502 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.685486 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 12 18:23:00.685581 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.685521 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-l2kpp\"" Jul 12 18:23:00.699265 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.699244 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd"] Jul 12 18:23:00.763136 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.763105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9506c9b2-3061-445c-ac23-dd18dc76c38d-tmp\") pod \"opendatahub-operator-controller-manager-db78fd4c7-w2mwd\" (UID: \"9506c9b2-3061-445c-ac23-dd18dc76c38d\") " pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:00.763288 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.763170 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9506c9b2-3061-445c-ac23-dd18dc76c38d-webhook-cert\") pod \"opendatahub-operator-controller-manager-db78fd4c7-w2mwd\" (UID: \"9506c9b2-3061-445c-ac23-dd18dc76c38d\") " pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:00.763288 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.763206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9506c9b2-3061-445c-ac23-dd18dc76c38d-manifests\") pod \"opendatahub-operator-controller-manager-db78fd4c7-w2mwd\" (UID: \"9506c9b2-3061-445c-ac23-dd18dc76c38d\") " pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:00.763288 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.763241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgpn7\" (UniqueName: \"kubernetes.io/projected/9506c9b2-3061-445c-ac23-dd18dc76c38d-kube-api-access-rgpn7\") pod \"opendatahub-operator-controller-manager-db78fd4c7-w2mwd\" (UID: \"9506c9b2-3061-445c-ac23-dd18dc76c38d\") " pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:00.763416 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.763287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9506c9b2-3061-445c-ac23-dd18dc76c38d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-db78fd4c7-w2mwd\" (UID: \"9506c9b2-3061-445c-ac23-dd18dc76c38d\") " pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:00.854673 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.854639 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:00.864414 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.864393 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9506c9b2-3061-445c-ac23-dd18dc76c38d-webhook-cert\") pod \"opendatahub-operator-controller-manager-db78fd4c7-w2mwd\" (UID: \"9506c9b2-3061-445c-ac23-dd18dc76c38d\") " pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:00.864533 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.864432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9506c9b2-3061-445c-ac23-dd18dc76c38d-manifests\") pod \"opendatahub-operator-controller-manager-db78fd4c7-w2mwd\" (UID: \"9506c9b2-3061-445c-ac23-dd18dc76c38d\") " pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:00.864533 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.864465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgpn7\" (UniqueName: \"kubernetes.io/projected/9506c9b2-3061-445c-ac23-dd18dc76c38d-kube-api-access-rgpn7\") pod \"opendatahub-operator-controller-manager-db78fd4c7-w2mwd\" (UID: \"9506c9b2-3061-445c-ac23-dd18dc76c38d\") " pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:00.864533 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.864493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9506c9b2-3061-445c-ac23-dd18dc76c38d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-db78fd4c7-w2mwd\" (UID: \"9506c9b2-3061-445c-ac23-dd18dc76c38d\") " pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:00.864646 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.864551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9506c9b2-3061-445c-ac23-dd18dc76c38d-tmp\") pod \"opendatahub-operator-controller-manager-db78fd4c7-w2mwd\" (UID: \"9506c9b2-3061-445c-ac23-dd18dc76c38d\") " pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:00.864906 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.864885 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9506c9b2-3061-445c-ac23-dd18dc76c38d-manifests\") pod \"opendatahub-operator-controller-manager-db78fd4c7-w2mwd\" (UID: \"9506c9b2-3061-445c-ac23-dd18dc76c38d\") " pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:00.866660 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.866640 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9506c9b2-3061-445c-ac23-dd18dc76c38d-tmp\") pod \"opendatahub-operator-controller-manager-db78fd4c7-w2mwd\" (UID: \"9506c9b2-3061-445c-ac23-dd18dc76c38d\") " pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:00.867020 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.866886 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9506c9b2-3061-445c-ac23-dd18dc76c38d-webhook-cert\") pod \"opendatahub-operator-controller-manager-db78fd4c7-w2mwd\" (UID: \"9506c9b2-3061-445c-ac23-dd18dc76c38d\") " pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:00.867020 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.866936 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9506c9b2-3061-445c-ac23-dd18dc76c38d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-db78fd4c7-w2mwd\" (UID: \"9506c9b2-3061-445c-ac23-dd18dc76c38d\") " pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:00.875607 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.875585 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgpn7\" (UniqueName: \"kubernetes.io/projected/9506c9b2-3061-445c-ac23-dd18dc76c38d-kube-api-access-rgpn7\") pod \"opendatahub-operator-controller-manager-db78fd4c7-w2mwd\" (UID: \"9506c9b2-3061-445c-ac23-dd18dc76c38d\") " pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:00.955049 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.954972 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:00.993082 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:00.993056 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:01.055615 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:01.055555 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:01.127155 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:01.127131 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd"] Jul 12 18:23:01.129756 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:23:01.129734 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9506c9b2_3061_445c_ac23_dd18dc76c38d.slice/crio-9146ee025c3b8198b2494433ec40a98991b660e101e3427c775d043c92529460 WatchSource:0}: Error finding container 9146ee025c3b8198b2494433ec40a98991b660e101e3427c775d043c92529460: Status 404 returned error can't find the container with id 9146ee025c3b8198b2494433ec40a98991b660e101e3427c775d043c92529460 Jul 12 18:23:01.131488 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:01.131469 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 18:23:01.134714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:01.134688 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:01.197385 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:01.197357 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:01.227554 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:01.227483 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" event={"ID":"9506c9b2-3061-445c-ac23-dd18dc76c38d","Type":"ContainerStarted","Data":"9146ee025c3b8198b2494433ec40a98991b660e101e3427c775d043c92529460"} Jul 12 18:23:01.300927 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:01.300886 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:01.403778 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:01.403746 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:04.242649 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:04.242617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" event={"ID":"9506c9b2-3061-445c-ac23-dd18dc76c38d","Type":"ContainerStarted","Data":"ec4438074864a0c0c3e7ba5c9ec73ff2745bfd22245eb10f53b0580d83df5152"} Jul 12 18:23:05.248048 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:05.248017 2576 generic.go:356] "Generic (PLEG): container finished" podID="9506c9b2-3061-445c-ac23-dd18dc76c38d" containerID="ec4438074864a0c0c3e7ba5c9ec73ff2745bfd22245eb10f53b0580d83df5152" exitCode=0 Jul 12 18:23:05.248390 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:05.248080 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" event={"ID":"9506c9b2-3061-445c-ac23-dd18dc76c38d","Type":"ContainerDied","Data":"ec4438074864a0c0c3e7ba5c9ec73ff2745bfd22245eb10f53b0580d83df5152"} Jul 12 18:23:06.255752 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:06.255710 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" event={"ID":"9506c9b2-3061-445c-ac23-dd18dc76c38d","Type":"ContainerStarted","Data":"66dcf05193395da445d8ab14fa8102189e567358a694f6c41712e50def37c308"} Jul 12 18:23:06.256261 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:06.256100 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:06.281414 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:06.281356 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" podStartSLOduration=1.8288885449999999 podStartE2EDuration="6.281337251s" totalImagesPullingTime="3.384348482s" totalInitContainerRuntime="1.068100224s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:23:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:23:06.279071228 +0000 UTC m=+348.046368811" watchObservedRunningTime="2026-07-12 18:23:06.281337251 +0000 UTC m=+348.048634833" Jul 12 18:23:06.312605 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:06.312572 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:06.417270 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:06.417235 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:12.339296 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:12.339220 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:12.434674 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:12.434631 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:17.261518 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:17.261489 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-db78fd4c7-w2mwd" Jul 12 18:23:19.463352 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:19.463316 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:19.636301 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:19.636266 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:20.041218 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.041179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:20.243915 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.243887 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7bd558f979-vxgtx"] Jul 12 18:23:20.251895 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.251858 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bd558f979-vxgtx" Jul 12 18:23:20.256041 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.256021 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 12 18:23:20.256507 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.256490 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9px9r\"" Jul 12 18:23:20.256601 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.256494 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 12 18:23:20.257261 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.257243 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 12 18:23:20.257404 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.257389 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 12 18:23:20.262257 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.262234 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bd558f979-vxgtx"] Jul 12 18:23:20.315672 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.315586 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07bec7be-8262-4463-94e7-a69a40481a95-tls-certs\") pod \"kube-auth-proxy-7bd558f979-vxgtx\" (UID: \"07bec7be-8262-4463-94e7-a69a40481a95\") " pod="openshift-ingress/kube-auth-proxy-7bd558f979-vxgtx" Jul 12 18:23:20.315672 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.315630 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c67m5\" (UniqueName: \"kubernetes.io/projected/07bec7be-8262-4463-94e7-a69a40481a95-kube-api-access-c67m5\") pod \"kube-auth-proxy-7bd558f979-vxgtx\" (UID: \"07bec7be-8262-4463-94e7-a69a40481a95\") " pod="openshift-ingress/kube-auth-proxy-7bd558f979-vxgtx" Jul 12 18:23:20.315672 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.315648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07bec7be-8262-4463-94e7-a69a40481a95-tmp\") pod \"kube-auth-proxy-7bd558f979-vxgtx\" (UID: \"07bec7be-8262-4463-94e7-a69a40481a95\") " pod="openshift-ingress/kube-auth-proxy-7bd558f979-vxgtx" Jul 12 18:23:20.336833 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.336779 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:20.416624 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.416591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c67m5\" (UniqueName: \"kubernetes.io/projected/07bec7be-8262-4463-94e7-a69a40481a95-kube-api-access-c67m5\") pod \"kube-auth-proxy-7bd558f979-vxgtx\" (UID: \"07bec7be-8262-4463-94e7-a69a40481a95\") " pod="openshift-ingress/kube-auth-proxy-7bd558f979-vxgtx" Jul 12 18:23:20.416840 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.416633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07bec7be-8262-4463-94e7-a69a40481a95-tmp\") pod \"kube-auth-proxy-7bd558f979-vxgtx\" (UID: \"07bec7be-8262-4463-94e7-a69a40481a95\") " pod="openshift-ingress/kube-auth-proxy-7bd558f979-vxgtx" Jul 12 18:23:20.416840 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.416702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07bec7be-8262-4463-94e7-a69a40481a95-tls-certs\") pod \"kube-auth-proxy-7bd558f979-vxgtx\" (UID: \"07bec7be-8262-4463-94e7-a69a40481a95\") " pod="openshift-ingress/kube-auth-proxy-7bd558f979-vxgtx" Jul 12 18:23:20.418802 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.418769 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07bec7be-8262-4463-94e7-a69a40481a95-tmp\") pod \"kube-auth-proxy-7bd558f979-vxgtx\" (UID: \"07bec7be-8262-4463-94e7-a69a40481a95\") " pod="openshift-ingress/kube-auth-proxy-7bd558f979-vxgtx" Jul 12 18:23:20.419065 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.419046 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07bec7be-8262-4463-94e7-a69a40481a95-tls-certs\") pod \"kube-auth-proxy-7bd558f979-vxgtx\" (UID: \"07bec7be-8262-4463-94e7-a69a40481a95\") " pod="openshift-ingress/kube-auth-proxy-7bd558f979-vxgtx" Jul 12 18:23:20.424681 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.424655 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c67m5\" (UniqueName: \"kubernetes.io/projected/07bec7be-8262-4463-94e7-a69a40481a95-kube-api-access-c67m5\") pod \"kube-auth-proxy-7bd558f979-vxgtx\" (UID: \"07bec7be-8262-4463-94e7-a69a40481a95\") " pod="openshift-ingress/kube-auth-proxy-7bd558f979-vxgtx" Jul 12 18:23:20.434688 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.434653 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:20.535511 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.535477 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:20.564584 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.564544 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bd558f979-vxgtx" Jul 12 18:23:20.701986 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:20.701962 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bd558f979-vxgtx"] Jul 12 18:23:20.704483 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:23:20.704454 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07bec7be_8262_4463_94e7_a69a40481a95.slice/crio-87139f4b5dea9ef81bbf403e9c79b9fb5e3a3890d20404fe5dd9ed4086760e98 WatchSource:0}: Error finding container 87139f4b5dea9ef81bbf403e9c79b9fb5e3a3890d20404fe5dd9ed4086760e98: Status 404 returned error can't find the container with id 87139f4b5dea9ef81bbf403e9c79b9fb5e3a3890d20404fe5dd9ed4086760e98 Jul 12 18:23:21.124475 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.124444 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2rln5"] Jul 12 18:23:21.127271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.127244 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2rln5" Jul 12 18:23:21.130077 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.130052 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 12 18:23:21.138959 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.138931 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2rln5"] Jul 12 18:23:21.143745 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.143719 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:21.224141 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.224109 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b72d8bae-1014-49ea-8f13-dff0f39c5ad2-tls-certs\") pod \"model-serving-api-54d89bd5fc-2rln5\" (UID: \"b72d8bae-1014-49ea-8f13-dff0f39c5ad2\") " pod="opendatahub/model-serving-api-54d89bd5fc-2rln5" Jul 12 18:23:21.224310 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.224147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnhqv\" (UniqueName: \"kubernetes.io/projected/b72d8bae-1014-49ea-8f13-dff0f39c5ad2-kube-api-access-dnhqv\") pod \"model-serving-api-54d89bd5fc-2rln5\" (UID: \"b72d8bae-1014-49ea-8f13-dff0f39c5ad2\") " pod="opendatahub/model-serving-api-54d89bd5fc-2rln5" Jul 12 18:23:21.316047 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.316014 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bd558f979-vxgtx" event={"ID":"07bec7be-8262-4463-94e7-a69a40481a95","Type":"ContainerStarted","Data":"87139f4b5dea9ef81bbf403e9c79b9fb5e3a3890d20404fe5dd9ed4086760e98"} Jul 12 18:23:21.325487 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.325443 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b72d8bae-1014-49ea-8f13-dff0f39c5ad2-tls-certs\") pod \"model-serving-api-54d89bd5fc-2rln5\" (UID: \"b72d8bae-1014-49ea-8f13-dff0f39c5ad2\") " pod="opendatahub/model-serving-api-54d89bd5fc-2rln5" Jul 12 18:23:21.325666 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.325511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnhqv\" (UniqueName: \"kubernetes.io/projected/b72d8bae-1014-49ea-8f13-dff0f39c5ad2-kube-api-access-dnhqv\") pod \"model-serving-api-54d89bd5fc-2rln5\" (UID: \"b72d8bae-1014-49ea-8f13-dff0f39c5ad2\") " pod="opendatahub/model-serving-api-54d89bd5fc-2rln5" Jul 12 18:23:21.325666 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:23:21.325606 2576 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 12 18:23:21.325832 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:23:21.325681 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72d8bae-1014-49ea-8f13-dff0f39c5ad2-tls-certs podName:b72d8bae-1014-49ea-8f13-dff0f39c5ad2 nodeName:}" failed. No retries permitted until 2026-07-12 18:23:21.825660108 +0000 UTC m=+363.592957668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b72d8bae-1014-49ea-8f13-dff0f39c5ad2-tls-certs") pod "model-serving-api-54d89bd5fc-2rln5" (UID: "b72d8bae-1014-49ea-8f13-dff0f39c5ad2") : secret "model-serving-api-tls" not found Jul 12 18:23:21.335561 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.335528 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:21.335706 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.335575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnhqv\" (UniqueName: \"kubernetes.io/projected/b72d8bae-1014-49ea-8f13-dff0f39c5ad2-kube-api-access-dnhqv\") pod \"model-serving-api-54d89bd5fc-2rln5\" (UID: \"b72d8bae-1014-49ea-8f13-dff0f39c5ad2\") " pod="opendatahub/model-serving-api-54d89bd5fc-2rln5" Jul 12 18:23:21.336544 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.336513 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf"] Jul 12 18:23:21.338843 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.338825 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.341267 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.341246 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 12 18:23:21.341374 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.341285 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 12 18:23:21.341374 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.341302 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 12 18:23:21.341483 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.341249 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 12 18:23:21.341483 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.341462 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-p96xq\"" Jul 12 18:23:21.354987 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.354968 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf"] Jul 12 18:23:21.396519 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.396443 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:21.426357 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.426329 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.426517 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.426363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v46sf\" (UniqueName: \"kubernetes.io/projected/109c2acf-dd82-4443-90df-6f88a0478cba-kube-api-access-v46sf\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.426517 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.426439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/109c2acf-dd82-4443-90df-6f88a0478cba-local-certs\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.426517 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.426506 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.426640 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.426548 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/109c2acf-dd82-4443-90df-6f88a0478cba-istio-token\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.426640 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.426603 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-cacerts\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.426640 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.426620 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/109c2acf-dd82-4443-90df-6f88a0478cba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.527638 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.527594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/109c2acf-dd82-4443-90df-6f88a0478cba-local-certs\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.527848 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.527665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.527848 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.527706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/109c2acf-dd82-4443-90df-6f88a0478cba-istio-token\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.527848 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.527779 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-cacerts\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.527848 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.527828 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/109c2acf-dd82-4443-90df-6f88a0478cba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.528404 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.528159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.528404 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.528217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v46sf\" (UniqueName: \"kubernetes.io/projected/109c2acf-dd82-4443-90df-6f88a0478cba-kube-api-access-v46sf\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.528582 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.528509 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/109c2acf-dd82-4443-90df-6f88a0478cba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.530530 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.530505 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/109c2acf-dd82-4443-90df-6f88a0478cba-local-certs\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.530735 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.530711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.531046 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.531024 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-cacerts\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.531223 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.531173 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.535984 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.535961 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v46sf\" (UniqueName: \"kubernetes.io/projected/109c2acf-dd82-4443-90df-6f88a0478cba-kube-api-access-v46sf\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.535984 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.535977 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/109c2acf-dd82-4443-90df-6f88a0478cba-istio-token\") pod \"istiod-openshift-gateway-7574459656-tv5bf\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.649874 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.649767 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:21.789434 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.789397 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:21.809933 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.809901 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf"] Jul 12 18:23:21.813475 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:23:21.813453 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod109c2acf_dd82_4443_90df_6f88a0478cba.slice/crio-5d58d7ceae1459bf539cd04661f846ca46574575fe3a1145a389978136ff4ff9 WatchSource:0}: Error finding container 5d58d7ceae1459bf539cd04661f846ca46574575fe3a1145a389978136ff4ff9: Status 404 returned error can't find the container with id 5d58d7ceae1459bf539cd04661f846ca46574575fe3a1145a389978136ff4ff9 Jul 12 18:23:21.830963 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.830936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b72d8bae-1014-49ea-8f13-dff0f39c5ad2-tls-certs\") pod \"model-serving-api-54d89bd5fc-2rln5\" (UID: \"b72d8bae-1014-49ea-8f13-dff0f39c5ad2\") " pod="opendatahub/model-serving-api-54d89bd5fc-2rln5" Jul 12 18:23:21.833681 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:21.833654 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b72d8bae-1014-49ea-8f13-dff0f39c5ad2-tls-certs\") pod \"model-serving-api-54d89bd5fc-2rln5\" (UID: \"b72d8bae-1014-49ea-8f13-dff0f39c5ad2\") " pod="opendatahub/model-serving-api-54d89bd5fc-2rln5" Jul 12 18:23:22.038506 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:22.038407 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2rln5" Jul 12 18:23:22.190060 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:22.189857 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2rln5"] Jul 12 18:23:22.193107 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:23:22.193082 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72d8bae_1014_49ea_8f13_dff0f39c5ad2.slice/crio-6db772c9d43d8ea17518bdbb6c49f39c15728e7508a6f66c847ee82986491c98 WatchSource:0}: Error finding container 6db772c9d43d8ea17518bdbb6c49f39c15728e7508a6f66c847ee82986491c98: Status 404 returned error can't find the container with id 6db772c9d43d8ea17518bdbb6c49f39c15728e7508a6f66c847ee82986491c98 Jul 12 18:23:22.213538 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:22.213486 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:22.321911 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:22.321823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2rln5" event={"ID":"b72d8bae-1014-49ea-8f13-dff0f39c5ad2","Type":"ContainerStarted","Data":"6db772c9d43d8ea17518bdbb6c49f39c15728e7508a6f66c847ee82986491c98"} Jul 12 18:23:22.329801 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:22.329734 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" event={"ID":"109c2acf-dd82-4443-90df-6f88a0478cba","Type":"ContainerStarted","Data":"5d58d7ceae1459bf539cd04661f846ca46574575fe3a1145a389978136ff4ff9"} Jul 12 18:23:23.344813 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:23.344551 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bd558f979-vxgtx" event={"ID":"07bec7be-8262-4463-94e7-a69a40481a95","Type":"ContainerStarted","Data":"4b676e36f1f9ac2fb4090e5f359eb5d250f7d8c1bde830e1c53c6aa803572a40"} Jul 12 18:23:25.260452 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:25.260421 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:23:25.260729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:25.260484 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:23:25.358833 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:25.358766 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2rln5" event={"ID":"b72d8bae-1014-49ea-8f13-dff0f39c5ad2","Type":"ContainerStarted","Data":"a0e8e7e42d0a88001e71bbe2c9bdd03e45930f0fb8c6dbf03f551f50946d0d96"} Jul 12 18:23:25.358993 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:25.358882 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-2rln5" Jul 12 18:23:25.360730 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:25.360702 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" event={"ID":"109c2acf-dd82-4443-90df-6f88a0478cba","Type":"ContainerStarted","Data":"86f6eb797829e28ce0296d3f55263d501522b0717ebf224d03b2defbcdd8ff30"} Jul 12 18:23:25.360885 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:25.360824 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:25.374743 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:25.374697 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7bd558f979-vxgtx" podStartSLOduration=3.064820916 podStartE2EDuration="5.374681423s" totalImagesPullingTime="2.309860507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:23:23.364510426 +0000 UTC m=+365.131808008" watchObservedRunningTime="2026-07-12 18:23:25.374681423 +0000 UTC m=+367.141979004" Jul 12 18:23:25.374980 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:25.374780 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-2rln5" podStartSLOduration=1.3158686689999999 podStartE2EDuration="4.37477412s" totalImagesPullingTime="3.058905451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:23:25.374260716 +0000 UTC m=+367.141558299" watchObservedRunningTime="2026-07-12 18:23:25.37477412 +0000 UTC m=+367.142071702" Jul 12 18:23:25.390411 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:25.390358 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" podStartSLOduration=0.945579068 podStartE2EDuration="4.390339651s" totalImagesPullingTime="3.444760583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:23:25.390326176 +0000 UTC m=+367.157623771" watchObservedRunningTime="2026-07-12 18:23:25.390339651 +0000 UTC m=+367.157637214" Jul 12 18:23:25.911254 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:25.911214 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:26.010198 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:26.010168 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:26.368838 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:26.368802 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-tv5bf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 12 18:23:26.369201 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:26.368863 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" podUID="109c2acf-dd82-4443-90df-6f88a0478cba" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 18:23:26.517794 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:26.517762 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:26.751649 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:26.751547 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:26.814222 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:26.814193 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:29.367578 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:29.367544 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:30.706314 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:30.706282 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:30.821249 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:30.821218 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z2545"] Jul 12 18:23:30.824738 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:30.824720 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z2545" Jul 12 18:23:30.827241 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:30.827221 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-mn4mc\"" Jul 12 18:23:30.828364 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:30.828349 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 12 18:23:30.842497 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:30.842470 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-c9sjr"] Jul 12 18:23:30.846966 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:30.846909 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:30.860607 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:30.860582 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z2545"] Jul 12 18:23:30.860738 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:30.860692 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c9sjr" Jul 12 18:23:30.863508 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:30.863485 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-nl6ld\"" Jul 12 18:23:30.863625 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:30.863508 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 12 18:23:30.869673 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:30.869650 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-c9sjr"] Jul 12 18:23:30.912803 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:30.912761 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh7hq\" (UniqueName: \"kubernetes.io/projected/e7b07d4c-cecc-49b7-8480-24163ba3fdb6-kube-api-access-jh7hq\") pod \"kserve-controller-manager-d6f4b65b8-z2545\" (UID: \"e7b07d4c-cecc-49b7-8480-24163ba3fdb6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z2545" Jul 12 18:23:30.912923 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:30.912812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7b07d4c-cecc-49b7-8480-24163ba3fdb6-cert\") pod \"kserve-controller-manager-d6f4b65b8-z2545\" (UID: \"e7b07d4c-cecc-49b7-8480-24163ba3fdb6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z2545" Jul 12 18:23:30.942034 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:30.942006 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:31.014049 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:31.013971 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25619e89-db09-4b28-b63c-1b5e432c719c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-c9sjr\" (UID: \"25619e89-db09-4b28-b63c-1b5e432c719c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c9sjr" Jul 12 18:23:31.014049 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:31.014016 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-558p7\" (UniqueName: \"kubernetes.io/projected/25619e89-db09-4b28-b63c-1b5e432c719c-kube-api-access-558p7\") pod \"llmisvc-controller-manager-777c6bd7d5-c9sjr\" (UID: \"25619e89-db09-4b28-b63c-1b5e432c719c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c9sjr" Jul 12 18:23:31.014208 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:31.014112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh7hq\" (UniqueName: \"kubernetes.io/projected/e7b07d4c-cecc-49b7-8480-24163ba3fdb6-kube-api-access-jh7hq\") pod \"kserve-controller-manager-d6f4b65b8-z2545\" (UID: \"e7b07d4c-cecc-49b7-8480-24163ba3fdb6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z2545" Jul 12 18:23:31.014208 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:31.014145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7b07d4c-cecc-49b7-8480-24163ba3fdb6-cert\") pod \"kserve-controller-manager-d6f4b65b8-z2545\" (UID: \"e7b07d4c-cecc-49b7-8480-24163ba3fdb6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z2545" Jul 12 18:23:31.016515 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:31.016487 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7b07d4c-cecc-49b7-8480-24163ba3fdb6-cert\") pod \"kserve-controller-manager-d6f4b65b8-z2545\" (UID: \"e7b07d4c-cecc-49b7-8480-24163ba3fdb6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z2545" Jul 12 18:23:31.026334 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:31.026314 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh7hq\" (UniqueName: \"kubernetes.io/projected/e7b07d4c-cecc-49b7-8480-24163ba3fdb6-kube-api-access-jh7hq\") pod \"kserve-controller-manager-d6f4b65b8-z2545\" (UID: \"e7b07d4c-cecc-49b7-8480-24163ba3fdb6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z2545" Jul 12 18:23:31.114748 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:31.114711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25619e89-db09-4b28-b63c-1b5e432c719c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-c9sjr\" (UID: \"25619e89-db09-4b28-b63c-1b5e432c719c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c9sjr" Jul 12 18:23:31.114913 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:31.114760 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-558p7\" (UniqueName: \"kubernetes.io/projected/25619e89-db09-4b28-b63c-1b5e432c719c-kube-api-access-558p7\") pod \"llmisvc-controller-manager-777c6bd7d5-c9sjr\" (UID: \"25619e89-db09-4b28-b63c-1b5e432c719c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c9sjr" Jul 12 18:23:31.117123 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:31.117096 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25619e89-db09-4b28-b63c-1b5e432c719c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-c9sjr\" (UID: \"25619e89-db09-4b28-b63c-1b5e432c719c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c9sjr" Jul 12 18:23:31.124679 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:31.124659 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-558p7\" (UniqueName: \"kubernetes.io/projected/25619e89-db09-4b28-b63c-1b5e432c719c-kube-api-access-558p7\") pod \"llmisvc-controller-manager-777c6bd7d5-c9sjr\" (UID: \"25619e89-db09-4b28-b63c-1b5e432c719c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c9sjr" Jul 12 18:23:31.135503 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:31.135481 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z2545" Jul 12 18:23:31.173422 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:31.173390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c9sjr" Jul 12 18:23:31.299177 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:31.299152 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z2545"] Jul 12 18:23:31.301627 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:23:31.301601 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b07d4c_cecc_49b7_8480_24163ba3fdb6.slice/crio-9f01ce024710d3f2430a56bab457b7201f8e9c55ebee4922a3a28273eda8b824 WatchSource:0}: Error finding container 9f01ce024710d3f2430a56bab457b7201f8e9c55ebee4922a3a28273eda8b824: Status 404 returned error can't find the container with id 9f01ce024710d3f2430a56bab457b7201f8e9c55ebee4922a3a28273eda8b824 Jul 12 18:23:31.331992 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:31.331965 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-c9sjr"] Jul 12 18:23:31.334537 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:23:31.334508 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod25619e89_db09_4b28_b63c_1b5e432c719c.slice/crio-44513bde031171c1cf4d80f85108df6a3f58b3731a9986d2a3b2eacf6512b165 WatchSource:0}: Error finding container 44513bde031171c1cf4d80f85108df6a3f58b3731a9986d2a3b2eacf6512b165: Status 404 returned error can't find the container with id 44513bde031171c1cf4d80f85108df6a3f58b3731a9986d2a3b2eacf6512b165 Jul 12 18:23:31.387492 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:31.387462 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c9sjr" event={"ID":"25619e89-db09-4b28-b63c-1b5e432c719c","Type":"ContainerStarted","Data":"44513bde031171c1cf4d80f85108df6a3f58b3731a9986d2a3b2eacf6512b165"} Jul 12 18:23:31.388521 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:31.388500 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z2545" event={"ID":"e7b07d4c-cecc-49b7-8480-24163ba3fdb6","Type":"ContainerStarted","Data":"9f01ce024710d3f2430a56bab457b7201f8e9c55ebee4922a3a28273eda8b824"} Jul 12 18:23:32.033215 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:32.033177 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:32.130463 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:32.130401 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:32.230032 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:32.230001 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:33.895168 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:33.895133 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf"] Jul 12 18:23:33.895823 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:33.895398 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" podUID="109c2acf-dd82-4443-90df-6f88a0478cba" containerName="discovery" containerID="cri-o://86f6eb797829e28ce0296d3f55263d501522b0717ebf224d03b2defbcdd8ff30" gracePeriod=30 Jul 12 18:23:34.294967 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.294942 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:34.404234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.404147 2576 generic.go:356] "Generic (PLEG): container finished" podID="109c2acf-dd82-4443-90df-6f88a0478cba" containerID="86f6eb797829e28ce0296d3f55263d501522b0717ebf224d03b2defbcdd8ff30" exitCode=0 Jul 12 18:23:34.404234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.404187 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" Jul 12 18:23:34.404457 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.404232 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" event={"ID":"109c2acf-dd82-4443-90df-6f88a0478cba","Type":"ContainerDied","Data":"86f6eb797829e28ce0296d3f55263d501522b0717ebf224d03b2defbcdd8ff30"} Jul 12 18:23:34.404457 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.404280 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf" event={"ID":"109c2acf-dd82-4443-90df-6f88a0478cba","Type":"ContainerDied","Data":"5d58d7ceae1459bf539cd04661f846ca46574575fe3a1145a389978136ff4ff9"} Jul 12 18:23:34.404457 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.404308 2576 scope.go:117] "RemoveContainer" containerID="86f6eb797829e28ce0296d3f55263d501522b0717ebf224d03b2defbcdd8ff30" Jul 12 18:23:34.405845 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.405824 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c9sjr" event={"ID":"25619e89-db09-4b28-b63c-1b5e432c719c","Type":"ContainerStarted","Data":"06885344211bdaca2e5e0c7dfbb93e92e0a693487ea9190fb20c58a38b6425c6"} Jul 12 18:23:34.405951 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.405897 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c9sjr" Jul 12 18:23:34.407334 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.407309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z2545" event={"ID":"e7b07d4c-cecc-49b7-8480-24163ba3fdb6","Type":"ContainerStarted","Data":"ce448330aedf8138f27eb8e20811c6db782b463e15a3268e12a76821513f43dc"} Jul 12 18:23:34.407436 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.407371 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z2545" Jul 12 18:23:34.413020 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.413001 2576 scope.go:117] "RemoveContainer" containerID="86f6eb797829e28ce0296d3f55263d501522b0717ebf224d03b2defbcdd8ff30" Jul 12 18:23:34.413256 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:23:34.413241 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f6eb797829e28ce0296d3f55263d501522b0717ebf224d03b2defbcdd8ff30\": container with ID starting with 86f6eb797829e28ce0296d3f55263d501522b0717ebf224d03b2defbcdd8ff30 not found: ID does not exist" containerID="86f6eb797829e28ce0296d3f55263d501522b0717ebf224d03b2defbcdd8ff30" Jul 12 18:23:34.413302 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.413263 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f6eb797829e28ce0296d3f55263d501522b0717ebf224d03b2defbcdd8ff30"} err="failed to get container status \"86f6eb797829e28ce0296d3f55263d501522b0717ebf224d03b2defbcdd8ff30\": rpc error: code = NotFound desc = could not find container \"86f6eb797829e28ce0296d3f55263d501522b0717ebf224d03b2defbcdd8ff30\": container with ID starting with 86f6eb797829e28ce0296d3f55263d501522b0717ebf224d03b2defbcdd8ff30 not found: ID does not exist" Jul 12 18:23:34.424689 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.424648 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c9sjr" podStartSLOduration=1.564776116 podStartE2EDuration="4.424634579s" totalImagesPullingTime="2.859858463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:23:34.423223704 +0000 UTC m=+376.190521287" watchObservedRunningTime="2026-07-12 18:23:34.424634579 +0000 UTC m=+376.191932160" Jul 12 18:23:34.440103 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.440060 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z2545" podStartSLOduration=2.217017365 podStartE2EDuration="4.440046329s" totalImagesPullingTime="2.223028964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:23:34.438824156 +0000 UTC m=+376.206121740" watchObservedRunningTime="2026-07-12 18:23:34.440046329 +0000 UTC m=+376.207343911" Jul 12 18:23:34.444305 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.444283 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-istio-csr-dns-cert\") pod \"109c2acf-dd82-4443-90df-6f88a0478cba\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " Jul 12 18:23:34.444393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.444310 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/109c2acf-dd82-4443-90df-6f88a0478cba-istio-token\") pod \"109c2acf-dd82-4443-90df-6f88a0478cba\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " Jul 12 18:23:34.444393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.444376 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/109c2acf-dd82-4443-90df-6f88a0478cba-istio-csr-ca-configmap\") pod \"109c2acf-dd82-4443-90df-6f88a0478cba\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " Jul 12 18:23:34.444492 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.444402 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-istio-kubeconfig\") pod \"109c2acf-dd82-4443-90df-6f88a0478cba\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " Jul 12 18:23:34.444492 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.444447 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/109c2acf-dd82-4443-90df-6f88a0478cba-local-certs\") pod \"109c2acf-dd82-4443-90df-6f88a0478cba\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " Jul 12 18:23:34.444598 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.444522 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v46sf\" (UniqueName: \"kubernetes.io/projected/109c2acf-dd82-4443-90df-6f88a0478cba-kube-api-access-v46sf\") pod \"109c2acf-dd82-4443-90df-6f88a0478cba\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " Jul 12 18:23:34.444598 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.444565 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-cacerts\") pod \"109c2acf-dd82-4443-90df-6f88a0478cba\" (UID: \"109c2acf-dd82-4443-90df-6f88a0478cba\") " Jul 12 18:23:34.445388 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.445358 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/109c2acf-dd82-4443-90df-6f88a0478cba-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "109c2acf-dd82-4443-90df-6f88a0478cba" (UID: "109c2acf-dd82-4443-90df-6f88a0478cba"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:23:34.446883 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.446852 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "109c2acf-dd82-4443-90df-6f88a0478cba" (UID: "109c2acf-dd82-4443-90df-6f88a0478cba"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:23:34.446982 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.446945 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/109c2acf-dd82-4443-90df-6f88a0478cba-istio-token" (OuterVolumeSpecName: "istio-token") pod "109c2acf-dd82-4443-90df-6f88a0478cba" (UID: "109c2acf-dd82-4443-90df-6f88a0478cba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:23:34.447050 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.447008 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "109c2acf-dd82-4443-90df-6f88a0478cba" (UID: "109c2acf-dd82-4443-90df-6f88a0478cba"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:23:34.447146 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.447126 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/109c2acf-dd82-4443-90df-6f88a0478cba-kube-api-access-v46sf" (OuterVolumeSpecName: "kube-api-access-v46sf") pod "109c2acf-dd82-4443-90df-6f88a0478cba" (UID: "109c2acf-dd82-4443-90df-6f88a0478cba"). InnerVolumeSpecName "kube-api-access-v46sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:23:34.447270 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.447248 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/109c2acf-dd82-4443-90df-6f88a0478cba-local-certs" (OuterVolumeSpecName: "local-certs") pod "109c2acf-dd82-4443-90df-6f88a0478cba" (UID: "109c2acf-dd82-4443-90df-6f88a0478cba"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:23:34.447353 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.447332 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-cacerts" (OuterVolumeSpecName: "cacerts") pod "109c2acf-dd82-4443-90df-6f88a0478cba" (UID: "109c2acf-dd82-4443-90df-6f88a0478cba"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:23:34.545758 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.545586 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/109c2acf-dd82-4443-90df-6f88a0478cba-istio-csr-ca-configmap\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:23:34.545758 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.545619 2576 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-istio-kubeconfig\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:23:34.545758 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.545633 2576 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/109c2acf-dd82-4443-90df-6f88a0478cba-local-certs\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:23:34.545758 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.545646 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v46sf\" (UniqueName: \"kubernetes.io/projected/109c2acf-dd82-4443-90df-6f88a0478cba-kube-api-access-v46sf\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:23:34.545758 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.545658 2576 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-cacerts\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:23:34.545758 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.545669 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/109c2acf-dd82-4443-90df-6f88a0478cba-istio-csr-dns-cert\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:23:34.545758 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.545681 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/109c2acf-dd82-4443-90df-6f88a0478cba-istio-token\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:23:34.724975 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.724943 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf"] Jul 12 18:23:34.728413 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.728385 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tv5bf"] Jul 12 18:23:34.817711 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:34.817671 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="109c2acf-dd82-4443-90df-6f88a0478cba" path="/var/lib/kubelet/pods/109c2acf-dd82-4443-90df-6f88a0478cba/volumes" Jul 12 18:23:36.371163 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:36.371135 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-2rln5" Jul 12 18:23:48.028434 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:48.028399 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:23:48.087266 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:23:48.087235 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:05.414155 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:05.414122 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-c9sjr" Jul 12 18:24:05.418285 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:05.418267 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z2545" Jul 12 18:24:20.975581 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:20.975550 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:21.074167 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:21.074141 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:21.111930 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:21.111896 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qb9vl"] Jul 12 18:24:21.112220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:21.112208 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="109c2acf-dd82-4443-90df-6f88a0478cba" containerName="discovery" Jul 12 18:24:21.112264 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:21.112222 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="109c2acf-dd82-4443-90df-6f88a0478cba" containerName="discovery" Jul 12 18:24:21.112298 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:21.112281 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="109c2acf-dd82-4443-90df-6f88a0478cba" containerName="discovery" Jul 12 18:24:21.114819 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:21.114804 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qb9vl" Jul 12 18:24:21.118563 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:21.118543 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 12 18:24:21.118563 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:21.118553 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 12 18:24:21.118727 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:21.118544 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-lmp6p\"" Jul 12 18:24:21.123087 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:21.123065 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qb9vl"] Jul 12 18:24:21.172081 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:21.172016 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:21.244081 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:21.243999 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ztk9\" (UniqueName: \"kubernetes.io/projected/2321885c-3e61-4307-bd59-9244b38c26e3-kube-api-access-6ztk9\") pod \"kuadrant-operator-catalog-qb9vl\" (UID: \"2321885c-3e61-4307-bd59-9244b38c26e3\") " pod="kuadrant-system/kuadrant-operator-catalog-qb9vl" Jul 12 18:24:21.344714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:21.344678 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ztk9\" (UniqueName: \"kubernetes.io/projected/2321885c-3e61-4307-bd59-9244b38c26e3-kube-api-access-6ztk9\") pod \"kuadrant-operator-catalog-qb9vl\" (UID: \"2321885c-3e61-4307-bd59-9244b38c26e3\") " pod="kuadrant-system/kuadrant-operator-catalog-qb9vl" Jul 12 18:24:21.356577 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:21.356554 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ztk9\" (UniqueName: \"kubernetes.io/projected/2321885c-3e61-4307-bd59-9244b38c26e3-kube-api-access-6ztk9\") pod \"kuadrant-operator-catalog-qb9vl\" (UID: \"2321885c-3e61-4307-bd59-9244b38c26e3\") " pod="kuadrant-system/kuadrant-operator-catalog-qb9vl" Jul 12 18:24:21.425291 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:21.425256 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qb9vl" Jul 12 18:24:21.549179 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:21.549008 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qb9vl"] Jul 12 18:24:21.551583 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:24:21.551555 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2321885c_3e61_4307_bd59_9244b38c26e3.slice/crio-2eb37916b188a1fa1867b81c5fe82ab4ac5288cd136a1ba8faa2337736f66de7 WatchSource:0}: Error finding container 2eb37916b188a1fa1867b81c5fe82ab4ac5288cd136a1ba8faa2337736f66de7: Status 404 returned error can't find the container with id 2eb37916b188a1fa1867b81c5fe82ab4ac5288cd136a1ba8faa2337736f66de7 Jul 12 18:24:21.606839 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:21.606810 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qb9vl" event={"ID":"2321885c-3e61-4307-bd59-9244b38c26e3","Type":"ContainerStarted","Data":"2eb37916b188a1fa1867b81c5fe82ab4ac5288cd136a1ba8faa2337736f66de7"} Jul 12 18:24:23.758736 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:23.758704 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:24.622127 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:24.622088 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qb9vl" event={"ID":"2321885c-3e61-4307-bd59-9244b38c26e3","Type":"ContainerStarted","Data":"b8185196ea78e6665d815f2d091bae186d467990c4872458cfe26a37c51a833a"} Jul 12 18:24:24.636960 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:24.636906 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-qb9vl" podStartSLOduration=1.5815396499999999 podStartE2EDuration="3.636890498s" totalImagesPullingTime="2.055350848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:24:24.635855482 +0000 UTC m=+426.403153065" watchObservedRunningTime="2026-07-12 18:24:24.636890498 +0000 UTC m=+426.404188083" Jul 12 18:24:25.806710 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:25.806676 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:25.861608 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:25.861578 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:27.095648 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:27.095615 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:31.426164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:31.426111 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-qb9vl" Jul 12 18:24:31.426617 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:31.426191 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-qb9vl" Jul 12 18:24:31.449030 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:31.448997 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-qb9vl" Jul 12 18:24:31.673907 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:31.673873 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-qb9vl" Jul 12 18:24:39.163570 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:39.163539 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:48.323621 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:48.323585 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:51.806605 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:51.806560 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:53.101200 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:53.101167 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc"] Jul 12 18:24:53.108960 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:53.108940 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:24:53.111347 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:53.111330 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gcg9j\"" Jul 12 18:24:53.116644 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:53.116619 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc"] Jul 12 18:24:53.202535 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:53.202497 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/698cb1c4-3254-4e8d-bba7-c8485525c6d6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" (UID: \"698cb1c4-3254-4e8d-bba7-c8485525c6d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:24:53.202701 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:53.202541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vplc9\" (UniqueName: \"kubernetes.io/projected/698cb1c4-3254-4e8d-bba7-c8485525c6d6-kube-api-access-vplc9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" (UID: \"698cb1c4-3254-4e8d-bba7-c8485525c6d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:24:53.303732 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:53.303694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/698cb1c4-3254-4e8d-bba7-c8485525c6d6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" (UID: \"698cb1c4-3254-4e8d-bba7-c8485525c6d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:24:53.303732 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:53.303739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vplc9\" (UniqueName: \"kubernetes.io/projected/698cb1c4-3254-4e8d-bba7-c8485525c6d6-kube-api-access-vplc9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" (UID: \"698cb1c4-3254-4e8d-bba7-c8485525c6d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:24:53.304114 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:53.304091 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/698cb1c4-3254-4e8d-bba7-c8485525c6d6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" (UID: \"698cb1c4-3254-4e8d-bba7-c8485525c6d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:24:53.314005 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:53.313977 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vplc9\" (UniqueName: \"kubernetes.io/projected/698cb1c4-3254-4e8d-bba7-c8485525c6d6-kube-api-access-vplc9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" (UID: \"698cb1c4-3254-4e8d-bba7-c8485525c6d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:24:53.419953 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:53.419853 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:24:53.551797 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:53.551631 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc"] Jul 12 18:24:53.554875 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:24:53.554844 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698cb1c4_3254_4e8d_bba7_c8485525c6d6.slice/crio-2b99600ed854d5268a682618a0eb159c6999ce0f22fb57f3047f6c9cf8380c9a WatchSource:0}: Error finding container 2b99600ed854d5268a682618a0eb159c6999ce0f22fb57f3047f6c9cf8380c9a: Status 404 returned error can't find the container with id 2b99600ed854d5268a682618a0eb159c6999ce0f22fb57f3047f6c9cf8380c9a Jul 12 18:24:53.733975 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:53.733899 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:53.745596 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:53.745567 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" event={"ID":"698cb1c4-3254-4e8d-bba7-c8485525c6d6","Type":"ContainerStarted","Data":"2b99600ed854d5268a682618a0eb159c6999ce0f22fb57f3047f6c9cf8380c9a"} Jul 12 18:24:54.992521 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:54.992487 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j"] Jul 12 18:24:54.996146 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:54.996126 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:24:54.999142 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:54.999113 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-59g5g\"" Jul 12 18:24:55.007413 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:55.007387 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j"] Jul 12 18:24:55.020192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:55.020165 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89nz6\" (UniqueName: \"kubernetes.io/projected/41da49e2-fd43-4602-a2a4-e12a6b858d80-kube-api-access-89nz6\") pod \"limitador-operator-controller-manager-859d7b78bd-vj25j\" (UID: \"41da49e2-fd43-4602-a2a4-e12a6b858d80\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:24:55.121370 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:55.121333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nz6\" (UniqueName: \"kubernetes.io/projected/41da49e2-fd43-4602-a2a4-e12a6b858d80-kube-api-access-89nz6\") pod \"limitador-operator-controller-manager-859d7b78bd-vj25j\" (UID: \"41da49e2-fd43-4602-a2a4-e12a6b858d80\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:24:55.132272 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:55.132226 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89nz6\" (UniqueName: \"kubernetes.io/projected/41da49e2-fd43-4602-a2a4-e12a6b858d80-kube-api-access-89nz6\") pod \"limitador-operator-controller-manager-859d7b78bd-vj25j\" (UID: \"41da49e2-fd43-4602-a2a4-e12a6b858d80\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:24:55.307018 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:55.306926 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:24:55.323088 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:55.323055 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:55.454273 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:55.454249 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j"] Jul 12 18:24:55.456922 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:24:55.456898 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41da49e2_fd43_4602_a2a4_e12a6b858d80.slice/crio-4d5eded62134417049b2ea22fee753ac8c7224810befeec04838890206153284 WatchSource:0}: Error finding container 4d5eded62134417049b2ea22fee753ac8c7224810befeec04838890206153284: Status 404 returned error can't find the container with id 4d5eded62134417049b2ea22fee753ac8c7224810befeec04838890206153284 Jul 12 18:24:55.726935 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:55.726896 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:55.757689 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:55.757652 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" event={"ID":"41da49e2-fd43-4602-a2a4-e12a6b858d80","Type":"ContainerStarted","Data":"4d5eded62134417049b2ea22fee753ac8c7224810befeec04838890206153284"} Jul 12 18:24:57.164479 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:57.164390 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:57.572656 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:57.572627 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-z9rk2"] Jul 12 18:24:57.576015 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:57.575998 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z9rk2" Jul 12 18:24:57.578704 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:57.578680 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4mq7z\"" Jul 12 18:24:57.586126 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:57.586101 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-z9rk2"] Jul 12 18:24:57.644416 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:57.644361 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d64kv\" (UniqueName: \"kubernetes.io/projected/a1a031af-1057-4bfc-aac0-a49cb91c4226-kube-api-access-d64kv\") pod \"authorino-operator-75cc8d5cb7-z9rk2\" (UID: \"a1a031af-1057-4bfc-aac0-a49cb91c4226\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-z9rk2" Jul 12 18:24:57.745073 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:57.745031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d64kv\" (UniqueName: \"kubernetes.io/projected/a1a031af-1057-4bfc-aac0-a49cb91c4226-kube-api-access-d64kv\") pod \"authorino-operator-75cc8d5cb7-z9rk2\" (UID: \"a1a031af-1057-4bfc-aac0-a49cb91c4226\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-z9rk2" Jul 12 18:24:57.760149 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:57.760088 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d64kv\" (UniqueName: \"kubernetes.io/projected/a1a031af-1057-4bfc-aac0-a49cb91c4226-kube-api-access-d64kv\") pod \"authorino-operator-75cc8d5cb7-z9rk2\" (UID: \"a1a031af-1057-4bfc-aac0-a49cb91c4226\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-z9rk2" Jul 12 18:24:57.773182 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:57.773146 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" event={"ID":"41da49e2-fd43-4602-a2a4-e12a6b858d80","Type":"ContainerStarted","Data":"5a335f5e6f398581ee594638fed16bc3608bbb5bba9e2fe41d775d8f76cdc51c"} Jul 12 18:24:57.773339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:57.773260 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:24:57.794991 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:57.794938 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" podStartSLOduration=2.348173377 podStartE2EDuration="3.794920088s" totalImagesPullingTime="1.446746711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:24:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:24:57.793837128 +0000 UTC m=+459.561134711" watchObservedRunningTime="2026-07-12 18:24:57.794920088 +0000 UTC m=+459.562217671" Jul 12 18:24:57.885858 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:57.885733 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z9rk2" Jul 12 18:24:58.779502 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:58.779457 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:58.870185 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:58.870149 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:58.990282 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:58.990215 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:58.999152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:58.999132 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-z9rk2"] Jul 12 18:24:59.090569 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:59.090538 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:59.516807 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:59.516687 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:59.616206 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:59.616176 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:24:59.786797 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:59.786688 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" event={"ID":"698cb1c4-3254-4e8d-bba7-c8485525c6d6","Type":"ContainerStarted","Data":"50c28c6ed55f8fc0126b7a3f86a5838f91b9bceb248ebe3b9bbd89496490d4e4"} Jul 12 18:24:59.787258 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:59.786822 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:24:59.788242 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:59.788208 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z9rk2" event={"ID":"a1a031af-1057-4bfc-aac0-a49cb91c4226","Type":"ContainerStarted","Data":"230afe549fb3f7e7b73d08ac4dc714aa01797246ff8666e8ebe05243afcf137a"} Jul 12 18:24:59.842044 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:24:59.841992 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" podStartSLOduration=1.470929773 podStartE2EDuration="6.84197647s" totalImagesPullingTime="5.371046697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:24:59.840375531 +0000 UTC m=+461.607673114" watchObservedRunningTime="2026-07-12 18:24:59.84197647 +0000 UTC m=+461.609274055" Jul 12 18:25:00.979381 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:00.979347 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-mx5pz"] Jul 12 18:25:00.987492 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:00.987467 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mx5pz" Jul 12 18:25:00.990018 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:00.989993 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 12 18:25:00.990142 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:00.990099 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-9976w\"" Jul 12 18:25:00.996621 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:00.996596 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-mx5pz"] Jul 12 18:25:01.079771 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:01.079715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg76g\" (UniqueName: \"kubernetes.io/projected/035b263e-40a1-4103-945d-81b47513d05e-kube-api-access-jg76g\") pod \"dns-operator-controller-manager-864dd78c4c-mx5pz\" (UID: \"035b263e-40a1-4103-945d-81b47513d05e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mx5pz" Jul 12 18:25:01.180318 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:01.180280 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg76g\" (UniqueName: \"kubernetes.io/projected/035b263e-40a1-4103-945d-81b47513d05e-kube-api-access-jg76g\") pod \"dns-operator-controller-manager-864dd78c4c-mx5pz\" (UID: \"035b263e-40a1-4103-945d-81b47513d05e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mx5pz" Jul 12 18:25:01.190478 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:01.190444 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg76g\" (UniqueName: \"kubernetes.io/projected/035b263e-40a1-4103-945d-81b47513d05e-kube-api-access-jg76g\") pod \"dns-operator-controller-manager-864dd78c4c-mx5pz\" (UID: \"035b263e-40a1-4103-945d-81b47513d05e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mx5pz" Jul 12 18:25:01.298737 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:01.298657 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mx5pz" Jul 12 18:25:01.445135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:01.445102 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-mx5pz"] Jul 12 18:25:01.448968 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:25:01.448940 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod035b263e_40a1_4103_945d_81b47513d05e.slice/crio-e474940c30722316dedb485f132382517c9ec2f3a16c994e268fc16609cb2624 WatchSource:0}: Error finding container e474940c30722316dedb485f132382517c9ec2f3a16c994e268fc16609cb2624: Status 404 returned error can't find the container with id e474940c30722316dedb485f132382517c9ec2f3a16c994e268fc16609cb2624 Jul 12 18:25:01.592465 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:01.592428 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:01.800496 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:01.800462 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z9rk2" event={"ID":"a1a031af-1057-4bfc-aac0-a49cb91c4226","Type":"ContainerStarted","Data":"86e72e44debb161010c063813b7d17e91b8c311dbd148b8d84c9ff6388cd71e0"} Jul 12 18:25:01.800673 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:01.800564 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z9rk2" Jul 12 18:25:01.801840 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:01.801818 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mx5pz" event={"ID":"035b263e-40a1-4103-945d-81b47513d05e","Type":"ContainerStarted","Data":"e474940c30722316dedb485f132382517c9ec2f3a16c994e268fc16609cb2624"} Jul 12 18:25:01.823152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:01.823105 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z9rk2" podStartSLOduration=2.5146854640000003 podStartE2EDuration="4.823089247s" totalImagesPullingTime="2.308403783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:25:01.822261698 +0000 UTC m=+463.589559279" watchObservedRunningTime="2026-07-12 18:25:01.823089247 +0000 UTC m=+463.590386829" Jul 12 18:25:02.141956 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.141921 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87"] Jul 12 18:25:02.145088 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.145069 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87" Jul 12 18:25:02.148311 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.148287 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-crn2p\"" Jul 12 18:25:02.148406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.148356 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 12 18:25:02.148465 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.148397 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 12 18:25:02.155890 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.155865 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87"] Jul 12 18:25:02.159410 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.159367 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:02.190666 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.190633 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xkn87\" (UID: \"e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87" Jul 12 18:25:02.190881 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.190683 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xkn87\" (UID: \"e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87" Jul 12 18:25:02.190881 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.190751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dg2c\" (UniqueName: \"kubernetes.io/projected/e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd-kube-api-access-7dg2c\") pod \"kuadrant-console-plugin-66b5774985-xkn87\" (UID: \"e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87" Jul 12 18:25:02.291567 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.291531 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dg2c\" (UniqueName: \"kubernetes.io/projected/e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd-kube-api-access-7dg2c\") pod \"kuadrant-console-plugin-66b5774985-xkn87\" (UID: \"e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87" Jul 12 18:25:02.291854 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.291641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xkn87\" (UID: \"e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87" Jul 12 18:25:02.291854 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.291683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xkn87\" (UID: \"e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87" Jul 12 18:25:02.291854 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:02.291822 2576 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 12 18:25:02.292038 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:02.291890 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd-plugin-serving-cert podName:e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd nodeName:}" failed. No retries permitted until 2026-07-12 18:25:02.791868359 +0000 UTC m=+464.559165922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-xkn87" (UID: "e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd") : secret "plugin-serving-cert" not found Jul 12 18:25:02.292420 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.292383 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xkn87\" (UID: \"e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87" Jul 12 18:25:02.299947 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.299920 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dg2c\" (UniqueName: \"kubernetes.io/projected/e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd-kube-api-access-7dg2c\") pod \"kuadrant-console-plugin-66b5774985-xkn87\" (UID: \"e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87" Jul 12 18:25:02.395387 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.395321 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:02.797244 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.797160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xkn87\" (UID: \"e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87" Jul 12 18:25:02.800144 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:02.800119 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xkn87\" (UID: \"e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87" Jul 12 18:25:03.066256 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:03.066162 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87" Jul 12 18:25:03.201419 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:03.201393 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87"] Jul 12 18:25:03.340322 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:03.340288 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:03.441549 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:03.441521 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:03.524058 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:25:03.524030 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1b304e6_edfe_4e6c_9bc7_df2a3ad7acdd.slice/crio-2eb139f6424b0151d0f7f5045650beeb46f59f0b30fc123c66a9e18b70dbcb18 WatchSource:0}: Error finding container 2eb139f6424b0151d0f7f5045650beeb46f59f0b30fc123c66a9e18b70dbcb18: Status 404 returned error can't find the container with id 2eb139f6424b0151d0f7f5045650beeb46f59f0b30fc123c66a9e18b70dbcb18 Jul 12 18:25:03.813630 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:03.813599 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87" event={"ID":"e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd","Type":"ContainerStarted","Data":"2eb139f6424b0151d0f7f5045650beeb46f59f0b30fc123c66a9e18b70dbcb18"} Jul 12 18:25:03.814854 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:03.814826 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mx5pz" event={"ID":"035b263e-40a1-4103-945d-81b47513d05e","Type":"ContainerStarted","Data":"47c87585cf4e5d255a561ef32d654eaef19df7bc4ba50b92f47c38c0678ec1d3"} Jul 12 18:25:03.814970 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:03.814943 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mx5pz" Jul 12 18:25:03.839197 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:03.838731 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mx5pz" podStartSLOduration=1.719091905 podStartE2EDuration="3.838711591s" totalImagesPullingTime="2.119619686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:25:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:25:03.836851043 +0000 UTC m=+465.604148626" watchObservedRunningTime="2026-07-12 18:25:03.838711591 +0000 UTC m=+465.606009174" Jul 12 18:25:08.781919 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:08.781883 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:25:09.505460 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:09.505400 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:09.606171 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:09.606125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:09.710548 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:09.710510 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:10.796614 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:10.796579 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:25:12.140539 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.140510 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:12.249124 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.249092 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:12.486244 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.486170 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc"] Jul 12 18:25:12.486424 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.486396 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" containerName="manager" containerID="cri-o://50c28c6ed55f8fc0126b7a3f86a5838f91b9bceb248ebe3b9bbd89496490d4e4" gracePeriod=2 Jul 12 18:25:12.488813 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:12.488762 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:25:12.489606 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.489571 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc"] Jul 12 18:25:12.501654 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.501627 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j"] Jul 12 18:25:12.501966 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.501933 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" podUID="41da49e2-fd43-4602-a2a4-e12a6b858d80" containerName="manager" containerID="cri-o://5a335f5e6f398581ee594638fed16bc3608bbb5bba9e2fe41d775d8f76cdc51c" gracePeriod=2 Jul 12 18:25:12.504190 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:12.504060 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:25:12.510608 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.510531 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd"] Jul 12 18:25:12.511238 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.511220 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" containerName="manager" Jul 12 18:25:12.511318 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.511242 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" containerName="manager" Jul 12 18:25:12.511378 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.511325 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" containerName="manager" Jul 12 18:25:12.514572 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.514549 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j"] Jul 12 18:25:12.514675 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.514661 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" Jul 12 18:25:12.517001 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:12.516921 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vj25j\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="41da49e2-fd43-4602-a2a4-e12a6b858d80" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:25:12.518992 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:12.518965 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:25:12.524164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.524139 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rcdg6"] Jul 12 18:25:12.524665 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.524647 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41da49e2-fd43-4602-a2a4-e12a6b858d80" containerName="manager" Jul 12 18:25:12.524746 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.524668 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="41da49e2-fd43-4602-a2a4-e12a6b858d80" containerName="manager" Jul 12 18:25:12.524819 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.524748 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="41da49e2-fd43-4602-a2a4-e12a6b858d80" containerName="manager" Jul 12 18:25:12.528341 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.528316 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd"] Jul 12 18:25:12.528444 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.528423 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rcdg6" Jul 12 18:25:12.538580 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.538543 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rcdg6"] Jul 12 18:25:12.567508 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:12.567468 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vj25j\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="41da49e2-fd43-4602-a2a4-e12a6b858d80" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:25:12.569400 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:12.569367 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:25:12.590899 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.590869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg6fx\" (UniqueName: \"kubernetes.io/projected/0bcdf6c4-6a25-4aed-adc7-a05f446a942f-kube-api-access-tg6fx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpkkd\" (UID: \"0bcdf6c4-6a25-4aed-adc7-a05f446a942f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" Jul 12 18:25:12.591036 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.590926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bcdf6c4-6a25-4aed-adc7-a05f446a942f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpkkd\" (UID: \"0bcdf6c4-6a25-4aed-adc7-a05f446a942f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" Jul 12 18:25:12.692128 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.692091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bcdf6c4-6a25-4aed-adc7-a05f446a942f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpkkd\" (UID: \"0bcdf6c4-6a25-4aed-adc7-a05f446a942f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" Jul 12 18:25:12.692128 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.692131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj6tv\" (UniqueName: \"kubernetes.io/projected/00a1e377-7d3a-4796-90c9-75f34b4daf18-kube-api-access-tj6tv\") pod \"limitador-operator-controller-manager-859d7b78bd-rcdg6\" (UID: \"00a1e377-7d3a-4796-90c9-75f34b4daf18\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rcdg6" Jul 12 18:25:12.692353 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.692232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg6fx\" (UniqueName: \"kubernetes.io/projected/0bcdf6c4-6a25-4aed-adc7-a05f446a942f-kube-api-access-tg6fx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpkkd\" (UID: \"0bcdf6c4-6a25-4aed-adc7-a05f446a942f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" Jul 12 18:25:12.692544 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.692523 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bcdf6c4-6a25-4aed-adc7-a05f446a942f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpkkd\" (UID: \"0bcdf6c4-6a25-4aed-adc7-a05f446a942f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" Jul 12 18:25:12.704482 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.704456 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg6fx\" (UniqueName: \"kubernetes.io/projected/0bcdf6c4-6a25-4aed-adc7-a05f446a942f-kube-api-access-tg6fx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vpkkd\" (UID: \"0bcdf6c4-6a25-4aed-adc7-a05f446a942f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" Jul 12 18:25:12.792804 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.792701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj6tv\" (UniqueName: \"kubernetes.io/projected/00a1e377-7d3a-4796-90c9-75f34b4daf18-kube-api-access-tj6tv\") pod \"limitador-operator-controller-manager-859d7b78bd-rcdg6\" (UID: \"00a1e377-7d3a-4796-90c9-75f34b4daf18\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rcdg6" Jul 12 18:25:12.809748 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.809712 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj6tv\" (UniqueName: \"kubernetes.io/projected/00a1e377-7d3a-4796-90c9-75f34b4daf18-kube-api-access-tj6tv\") pod \"limitador-operator-controller-manager-859d7b78bd-rcdg6\" (UID: \"00a1e377-7d3a-4796-90c9-75f34b4daf18\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rcdg6" Jul 12 18:25:12.811354 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.811329 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z9rk2" Jul 12 18:25:12.813794 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:12.813747 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vj25j\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="41da49e2-fd43-4602-a2a4-e12a6b858d80" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:25:12.835558 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:12.835506 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:25:12.926742 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.926707 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" Jul 12 18:25:12.934311 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:12.934222 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rcdg6" Jul 12 18:25:13.652409 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:13.652376 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh"] Jul 12 18:25:13.665601 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:13.665574 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" Jul 12 18:25:13.676530 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:13.676035 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh"] Jul 12 18:25:13.802520 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:13.802470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1e93a66-863f-45c1-9e29-01d6e22cb168-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-sc6kh\" (UID: \"a1e93a66-863f-45c1-9e29-01d6e22cb168\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" Jul 12 18:25:13.802520 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:13.802522 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5q77\" (UniqueName: \"kubernetes.io/projected/a1e93a66-863f-45c1-9e29-01d6e22cb168-kube-api-access-c5q77\") pod \"kuadrant-operator-controller-manager-86878c769f-sc6kh\" (UID: \"a1e93a66-863f-45c1-9e29-01d6e22cb168\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" Jul 12 18:25:13.904096 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:13.904011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1e93a66-863f-45c1-9e29-01d6e22cb168-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-sc6kh\" (UID: \"a1e93a66-863f-45c1-9e29-01d6e22cb168\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" Jul 12 18:25:13.904096 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:13.904054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5q77\" (UniqueName: \"kubernetes.io/projected/a1e93a66-863f-45c1-9e29-01d6e22cb168-kube-api-access-c5q77\") pod \"kuadrant-operator-controller-manager-86878c769f-sc6kh\" (UID: \"a1e93a66-863f-45c1-9e29-01d6e22cb168\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" Jul 12 18:25:13.904501 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:13.904478 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1e93a66-863f-45c1-9e29-01d6e22cb168-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-sc6kh\" (UID: \"a1e93a66-863f-45c1-9e29-01d6e22cb168\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" Jul 12 18:25:13.918808 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:13.918761 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5q77\" (UniqueName: \"kubernetes.io/projected/a1e93a66-863f-45c1-9e29-01d6e22cb168-kube-api-access-c5q77\") pod \"kuadrant-operator-controller-manager-86878c769f-sc6kh\" (UID: \"a1e93a66-863f-45c1-9e29-01d6e22cb168\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" Jul 12 18:25:13.980471 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:13.980431 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" Jul 12 18:25:14.821971 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:14.821944 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mx5pz" Jul 12 18:25:16.734978 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:16.734930 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:16.801008 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:16.800975 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:17.141042 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:17.141012 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:17.203274 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:17.203238 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:18.794753 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:18.794715 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:18.850704 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:18.850661 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:21.334074 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:21.334041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:21.402357 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:21.402321 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:24.346467 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.346428 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:24.448059 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.448027 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:24.881656 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.881333 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:25:24.884084 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:24.884052 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:25:24.884566 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.884547 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:25:24.886866 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:24.886817 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:25:24.891170 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:24.891139 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vj25j\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="41da49e2-fd43-4602-a2a4-e12a6b858d80" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:25:24.919699 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.919672 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd"] Jul 12 18:25:24.925129 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:25:24.924721 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcdf6c4_6a25_4aed_adc7_a05f446a942f.slice/crio-587bb0e4d198c2440f84144fa74687864d729b640ba66aad1ba073d65241497a WatchSource:0}: Error finding container 587bb0e4d198c2440f84144fa74687864d729b640ba66aad1ba073d65241497a: Status 404 returned error can't find the container with id 587bb0e4d198c2440f84144fa74687864d729b640ba66aad1ba073d65241497a Jul 12 18:25:24.934895 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.934860 2576 generic.go:356] "Generic (PLEG): container finished" podID="41da49e2-fd43-4602-a2a4-e12a6b858d80" containerID="5a335f5e6f398581ee594638fed16bc3608bbb5bba9e2fe41d775d8f76cdc51c" exitCode=0 Jul 12 18:25:24.935090 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.934906 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:25:24.936401 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.935908 2576 scope.go:117] "RemoveContainer" containerID="5a335f5e6f398581ee594638fed16bc3608bbb5bba9e2fe41d775d8f76cdc51c" Jul 12 18:25:24.937824 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.937651 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" event={"ID":"0bcdf6c4-6a25-4aed-adc7-a05f446a942f","Type":"ContainerStarted","Data":"587bb0e4d198c2440f84144fa74687864d729b640ba66aad1ba073d65241497a"} Jul 12 18:25:24.937900 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:24.937840 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:25:24.938765 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.938710 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh"] Jul 12 18:25:24.939477 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.939452 2576 generic.go:356] "Generic (PLEG): container finished" podID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" containerID="50c28c6ed55f8fc0126b7a3f86a5838f91b9bceb248ebe3b9bbd89496490d4e4" exitCode=0 Jul 12 18:25:24.939584 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.939504 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:25:24.939992 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:24.939963 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vj25j\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="41da49e2-fd43-4602-a2a4-e12a6b858d80" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:25:24.942063 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:24.942041 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vj25j\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="41da49e2-fd43-4602-a2a4-e12a6b858d80" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:25:24.944031 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:24.943996 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:25:24.944996 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:25:24.944946 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e93a66_863f_45c1_9e29_01d6e22cb168.slice/crio-3d7508b43853264ac293036f631ae2e4597c7d356c8417240b6160b59fc2485b WatchSource:0}: Error finding container 3d7508b43853264ac293036f631ae2e4597c7d356c8417240b6160b59fc2485b: Status 404 returned error can't find the container with id 3d7508b43853264ac293036f631ae2e4597c7d356c8417240b6160b59fc2485b Jul 12 18:25:24.950052 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.949977 2576 scope.go:117] "RemoveContainer" containerID="5a335f5e6f398581ee594638fed16bc3608bbb5bba9e2fe41d775d8f76cdc51c" Jul 12 18:25:24.950558 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:24.950522 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a335f5e6f398581ee594638fed16bc3608bbb5bba9e2fe41d775d8f76cdc51c\": container with ID starting with 5a335f5e6f398581ee594638fed16bc3608bbb5bba9e2fe41d775d8f76cdc51c not found: ID does not exist" containerID="5a335f5e6f398581ee594638fed16bc3608bbb5bba9e2fe41d775d8f76cdc51c" Jul 12 18:25:24.950624 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.950559 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a335f5e6f398581ee594638fed16bc3608bbb5bba9e2fe41d775d8f76cdc51c"} err="failed to get container status \"5a335f5e6f398581ee594638fed16bc3608bbb5bba9e2fe41d775d8f76cdc51c\": rpc error: code = NotFound desc = could not find container \"5a335f5e6f398581ee594638fed16bc3608bbb5bba9e2fe41d775d8f76cdc51c\": container with ID starting with 5a335f5e6f398581ee594638fed16bc3608bbb5bba9e2fe41d775d8f76cdc51c not found: ID does not exist" Jul 12 18:25:24.950624 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.950582 2576 scope.go:117] "RemoveContainer" containerID="50c28c6ed55f8fc0126b7a3f86a5838f91b9bceb248ebe3b9bbd89496490d4e4" Jul 12 18:25:24.962024 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.961828 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rcdg6"] Jul 12 18:25:24.964606 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:25:24.964580 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a1e377_7d3a_4796_90c9_75f34b4daf18.slice/crio-f7ad72c3c9a7df0a7ca841878ec53bbcd2402e7e563d27bfd0b5ac17818bb40e WatchSource:0}: Error finding container f7ad72c3c9a7df0a7ca841878ec53bbcd2402e7e563d27bfd0b5ac17818bb40e: Status 404 returned error can't find the container with id f7ad72c3c9a7df0a7ca841878ec53bbcd2402e7e563d27bfd0b5ac17818bb40e Jul 12 18:25:24.967733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.967711 2576 scope.go:117] "RemoveContainer" containerID="50c28c6ed55f8fc0126b7a3f86a5838f91b9bceb248ebe3b9bbd89496490d4e4" Jul 12 18:25:24.968164 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:24.968118 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50c28c6ed55f8fc0126b7a3f86a5838f91b9bceb248ebe3b9bbd89496490d4e4\": container with ID starting with 50c28c6ed55f8fc0126b7a3f86a5838f91b9bceb248ebe3b9bbd89496490d4e4 not found: ID does not exist" containerID="50c28c6ed55f8fc0126b7a3f86a5838f91b9bceb248ebe3b9bbd89496490d4e4" Jul 12 18:25:24.968230 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:24.968157 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50c28c6ed55f8fc0126b7a3f86a5838f91b9bceb248ebe3b9bbd89496490d4e4"} err="failed to get container status \"50c28c6ed55f8fc0126b7a3f86a5838f91b9bceb248ebe3b9bbd89496490d4e4\": rpc error: code = NotFound desc = could not find container \"50c28c6ed55f8fc0126b7a3f86a5838f91b9bceb248ebe3b9bbd89496490d4e4\": container with ID starting with 50c28c6ed55f8fc0126b7a3f86a5838f91b9bceb248ebe3b9bbd89496490d4e4 not found: ID does not exist" Jul 12 18:25:25.007707 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.007683 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89nz6\" (UniqueName: \"kubernetes.io/projected/41da49e2-fd43-4602-a2a4-e12a6b858d80-kube-api-access-89nz6\") pod \"41da49e2-fd43-4602-a2a4-e12a6b858d80\" (UID: \"41da49e2-fd43-4602-a2a4-e12a6b858d80\") " Jul 12 18:25:25.007840 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.007797 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/698cb1c4-3254-4e8d-bba7-c8485525c6d6-extensions-socket-volume\") pod \"698cb1c4-3254-4e8d-bba7-c8485525c6d6\" (UID: \"698cb1c4-3254-4e8d-bba7-c8485525c6d6\") " Jul 12 18:25:25.007907 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.007846 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vplc9\" (UniqueName: \"kubernetes.io/projected/698cb1c4-3254-4e8d-bba7-c8485525c6d6-kube-api-access-vplc9\") pod \"698cb1c4-3254-4e8d-bba7-c8485525c6d6\" (UID: \"698cb1c4-3254-4e8d-bba7-c8485525c6d6\") " Jul 12 18:25:25.008289 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.008267 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698cb1c4-3254-4e8d-bba7-c8485525c6d6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "698cb1c4-3254-4e8d-bba7-c8485525c6d6" (UID: "698cb1c4-3254-4e8d-bba7-c8485525c6d6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:25:25.010314 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.010284 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698cb1c4-3254-4e8d-bba7-c8485525c6d6-kube-api-access-vplc9" (OuterVolumeSpecName: "kube-api-access-vplc9") pod "698cb1c4-3254-4e8d-bba7-c8485525c6d6" (UID: "698cb1c4-3254-4e8d-bba7-c8485525c6d6"). InnerVolumeSpecName "kube-api-access-vplc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:25:25.010452 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.010432 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41da49e2-fd43-4602-a2a4-e12a6b858d80-kube-api-access-89nz6" (OuterVolumeSpecName: "kube-api-access-89nz6") pod "41da49e2-fd43-4602-a2a4-e12a6b858d80" (UID: "41da49e2-fd43-4602-a2a4-e12a6b858d80"). InnerVolumeSpecName "kube-api-access-89nz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:25:25.108613 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.108585 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89nz6\" (UniqueName: \"kubernetes.io/projected/41da49e2-fd43-4602-a2a4-e12a6b858d80-kube-api-access-89nz6\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:25:25.108613 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.108610 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/698cb1c4-3254-4e8d-bba7-c8485525c6d6-extensions-socket-volume\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:25:25.108613 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.108620 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vplc9\" (UniqueName: \"kubernetes.io/projected/698cb1c4-3254-4e8d-bba7-c8485525c6d6-kube-api-access-vplc9\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:25:25.250296 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:25.250251 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vj25j\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="41da49e2-fd43-4602-a2a4-e12a6b858d80" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:25:25.252827 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:25.252758 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:25:25.255224 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:25.255194 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:25:25.257227 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:25.257202 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vj25j\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="41da49e2-fd43-4602-a2a4-e12a6b858d80" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:25:25.948070 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.947962 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" event={"ID":"0bcdf6c4-6a25-4aed-adc7-a05f446a942f","Type":"ContainerStarted","Data":"559d4b3e112807263a7b7e7d67c9faf79b9adba690f04aaf4d616db69af33d3e"} Jul 12 18:25:25.948476 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.948072 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" Jul 12 18:25:25.949524 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.949498 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87" event={"ID":"e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd","Type":"ContainerStarted","Data":"ad0f3ed0304c6838f7a54ab5f83c8122dc072538e3f535c5e0553ff6ee0291de"} Jul 12 18:25:25.950845 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:25.950819 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vj25j\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="41da49e2-fd43-4602-a2a4-e12a6b858d80" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:25:25.951204 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.951177 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rcdg6" event={"ID":"00a1e377-7d3a-4796-90c9-75f34b4daf18","Type":"ContainerStarted","Data":"c44dcb38664ea91926043a6892082fce4ff2afe3f2c38fe3525ac90fb08ce7c7"} Jul 12 18:25:25.951292 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.951227 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rcdg6" Jul 12 18:25:25.951292 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.951251 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rcdg6" event={"ID":"00a1e377-7d3a-4796-90c9-75f34b4daf18","Type":"ContainerStarted","Data":"f7ad72c3c9a7df0a7ca841878ec53bbcd2402e7e563d27bfd0b5ac17818bb40e"} Jul 12 18:25:25.952747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.952727 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" event={"ID":"a1e93a66-863f-45c1-9e29-01d6e22cb168","Type":"ContainerStarted","Data":"eccf4a1df5e98d212935efeff6f12d7b332a42c808faaa8ee44ac75441c10e2a"} Jul 12 18:25:25.952873 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.952771 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" event={"ID":"a1e93a66-863f-45c1-9e29-01d6e22cb168","Type":"ContainerStarted","Data":"3d7508b43853264ac293036f631ae2e4597c7d356c8417240b6160b59fc2485b"} Jul 12 18:25:25.952873 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.952835 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" Jul 12 18:25:25.952965 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:25.952880 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:25:25.977953 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:25.977910 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dngxc\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dngxc" Jul 12 18:25:25.978621 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.978359 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" podStartSLOduration=13.978342262 podStartE2EDuration="13.978342262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:25:25.974986759 +0000 UTC m=+487.742284342" watchObservedRunningTime="2026-07-12 18:25:25.978342262 +0000 UTC m=+487.745639845" Jul 12 18:25:25.997914 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:25.997869 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rcdg6" podStartSLOduration=13.99785468 podStartE2EDuration="13.99785468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:25:25.997105201 +0000 UTC m=+487.764402785" watchObservedRunningTime="2026-07-12 18:25:25.99785468 +0000 UTC m=+487.765152262" Jul 12 18:25:25.999069 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:25.999038 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vj25j\" is forbidden: User \"system:node:ip-10-0-132-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-5.ec2.internal' and this object" podUID="41da49e2-fd43-4602-a2a4-e12a6b858d80" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vj25j" Jul 12 18:25:26.020347 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:26.020297 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xkn87" podStartSLOduration=2.651854462 podStartE2EDuration="24.02028331s" totalImagesPullingTime="21.368428848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:25:26.018729163 +0000 UTC m=+487.786026739" watchObservedRunningTime="2026-07-12 18:25:26.02028331 +0000 UTC m=+487.787580891" Jul 12 18:25:26.037981 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:26.037933 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" podStartSLOduration=13.037912307 podStartE2EDuration="13.037912307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:25:26.036886446 +0000 UTC m=+487.804184028" watchObservedRunningTime="2026-07-12 18:25:26.037912307 +0000 UTC m=+487.805209889" Jul 12 18:25:26.816998 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:26.816968 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41da49e2-fd43-4602-a2a4-e12a6b858d80" path="/var/lib/kubelet/pods/41da49e2-fd43-4602-a2a4-e12a6b858d80/volumes" Jul 12 18:25:26.817325 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:26.817312 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="698cb1c4-3254-4e8d-bba7-c8485525c6d6" path="/var/lib/kubelet/pods/698cb1c4-3254-4e8d-bba7-c8485525c6d6/volumes" Jul 12 18:25:31.275717 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:31.275678 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:31.377107 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:31.377074 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:36.075556 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:36.075524 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:36.174622 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:36.174592 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:36.960906 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:36.960880 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" Jul 12 18:25:36.961071 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:36.960945 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" Jul 12 18:25:36.961071 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:36.961067 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rcdg6" Jul 12 18:25:37.053805 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:37.053753 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd"] Jul 12 18:25:37.054081 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:37.054053 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" podUID="0bcdf6c4-6a25-4aed-adc7-a05f446a942f" containerName="manager" containerID="cri-o://559d4b3e112807263a7b7e7d67c9faf79b9adba690f04aaf4d616db69af33d3e" gracePeriod=10 Jul 12 18:25:37.831237 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:37.831212 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" Jul 12 18:25:37.918119 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:37.918085 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bcdf6c4-6a25-4aed-adc7-a05f446a942f-extensions-socket-volume\") pod \"0bcdf6c4-6a25-4aed-adc7-a05f446a942f\" (UID: \"0bcdf6c4-6a25-4aed-adc7-a05f446a942f\") " Jul 12 18:25:37.918278 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:37.918149 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg6fx\" (UniqueName: \"kubernetes.io/projected/0bcdf6c4-6a25-4aed-adc7-a05f446a942f-kube-api-access-tg6fx\") pod \"0bcdf6c4-6a25-4aed-adc7-a05f446a942f\" (UID: \"0bcdf6c4-6a25-4aed-adc7-a05f446a942f\") " Jul 12 18:25:37.918501 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:37.918481 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bcdf6c4-6a25-4aed-adc7-a05f446a942f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0bcdf6c4-6a25-4aed-adc7-a05f446a942f" (UID: "0bcdf6c4-6a25-4aed-adc7-a05f446a942f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:25:37.920149 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:37.920123 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bcdf6c4-6a25-4aed-adc7-a05f446a942f-kube-api-access-tg6fx" (OuterVolumeSpecName: "kube-api-access-tg6fx") pod "0bcdf6c4-6a25-4aed-adc7-a05f446a942f" (UID: "0bcdf6c4-6a25-4aed-adc7-a05f446a942f"). InnerVolumeSpecName "kube-api-access-tg6fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:25:38.014773 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:38.014684 2576 generic.go:356] "Generic (PLEG): container finished" podID="0bcdf6c4-6a25-4aed-adc7-a05f446a942f" containerID="559d4b3e112807263a7b7e7d67c9faf79b9adba690f04aaf4d616db69af33d3e" exitCode=0 Jul 12 18:25:38.014773 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:38.014728 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" Jul 12 18:25:38.014773 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:38.014748 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" event={"ID":"0bcdf6c4-6a25-4aed-adc7-a05f446a942f","Type":"ContainerDied","Data":"559d4b3e112807263a7b7e7d67c9faf79b9adba690f04aaf4d616db69af33d3e"} Jul 12 18:25:38.015061 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:38.014835 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd" event={"ID":"0bcdf6c4-6a25-4aed-adc7-a05f446a942f","Type":"ContainerDied","Data":"587bb0e4d198c2440f84144fa74687864d729b640ba66aad1ba073d65241497a"} Jul 12 18:25:38.015061 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:38.014866 2576 scope.go:117] "RemoveContainer" containerID="559d4b3e112807263a7b7e7d67c9faf79b9adba690f04aaf4d616db69af33d3e" Jul 12 18:25:38.018717 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:38.018695 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bcdf6c4-6a25-4aed-adc7-a05f446a942f-extensions-socket-volume\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:25:38.018821 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:38.018720 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg6fx\" (UniqueName: \"kubernetes.io/projected/0bcdf6c4-6a25-4aed-adc7-a05f446a942f-kube-api-access-tg6fx\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:25:38.023553 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:38.023534 2576 scope.go:117] "RemoveContainer" containerID="559d4b3e112807263a7b7e7d67c9faf79b9adba690f04aaf4d616db69af33d3e" Jul 12 18:25:38.023816 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:25:38.023779 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559d4b3e112807263a7b7e7d67c9faf79b9adba690f04aaf4d616db69af33d3e\": container with ID starting with 559d4b3e112807263a7b7e7d67c9faf79b9adba690f04aaf4d616db69af33d3e not found: ID does not exist" containerID="559d4b3e112807263a7b7e7d67c9faf79b9adba690f04aaf4d616db69af33d3e" Jul 12 18:25:38.023876 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:38.023837 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559d4b3e112807263a7b7e7d67c9faf79b9adba690f04aaf4d616db69af33d3e"} err="failed to get container status \"559d4b3e112807263a7b7e7d67c9faf79b9adba690f04aaf4d616db69af33d3e\": rpc error: code = NotFound desc = could not find container \"559d4b3e112807263a7b7e7d67c9faf79b9adba690f04aaf4d616db69af33d3e\": container with ID starting with 559d4b3e112807263a7b7e7d67c9faf79b9adba690f04aaf4d616db69af33d3e not found: ID does not exist" Jul 12 18:25:38.042506 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:38.042474 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd"] Jul 12 18:25:38.046676 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:38.046654 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vpkkd"] Jul 12 18:25:38.548289 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:38.548252 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:38.644806 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:38.644760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:38.817863 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:38.817769 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bcdf6c4-6a25-4aed-adc7-a05f446a942f" path="/var/lib/kubelet/pods/0bcdf6c4-6a25-4aed-adc7-a05f446a942f/volumes" Jul 12 18:25:44.284991 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:44.284956 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:44.385603 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:44.385570 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:52.070212 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:52.070157 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:52.171662 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:52.171627 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:53.407947 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.407911 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x"] Jul 12 18:25:53.408463 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.408442 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bcdf6c4-6a25-4aed-adc7-a05f446a942f" containerName="manager" Jul 12 18:25:53.408544 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.408466 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bcdf6c4-6a25-4aed-adc7-a05f446a942f" containerName="manager" Jul 12 18:25:53.408739 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.408577 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bcdf6c4-6a25-4aed-adc7-a05f446a942f" containerName="manager" Jul 12 18:25:53.416886 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.416862 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.419621 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.419587 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 12 18:25:53.419876 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.419856 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hpc64\"" Jul 12 18:25:53.422992 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.422968 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x"] Jul 12 18:25:53.441309 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.441278 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:53.554985 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.554951 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678cd7d5-7144-400c-a44f-c3b07b36fab9-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.555158 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.554999 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678cd7d5-7144-400c-a44f-c3b07b36fab9-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.555158 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.555037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678cd7d5-7144-400c-a44f-c3b07b36fab9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.555158 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.555107 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9fvh\" (UniqueName: \"kubernetes.io/projected/678cd7d5-7144-400c-a44f-c3b07b36fab9-kube-api-access-j9fvh\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.555158 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.555144 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678cd7d5-7144-400c-a44f-c3b07b36fab9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.555296 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.555161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/678cd7d5-7144-400c-a44f-c3b07b36fab9-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.555296 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.555219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678cd7d5-7144-400c-a44f-c3b07b36fab9-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.555296 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.555243 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678cd7d5-7144-400c-a44f-c3b07b36fab9-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.555296 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.555268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678cd7d5-7144-400c-a44f-c3b07b36fab9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.656104 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.656051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678cd7d5-7144-400c-a44f-c3b07b36fab9-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.656290 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.656138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678cd7d5-7144-400c-a44f-c3b07b36fab9-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.656290 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.656209 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678cd7d5-7144-400c-a44f-c3b07b36fab9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.656290 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.656268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9fvh\" (UniqueName: \"kubernetes.io/projected/678cd7d5-7144-400c-a44f-c3b07b36fab9-kube-api-access-j9fvh\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.656452 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.656328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678cd7d5-7144-400c-a44f-c3b07b36fab9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.656452 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.656359 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/678cd7d5-7144-400c-a44f-c3b07b36fab9-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.656452 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.656399 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678cd7d5-7144-400c-a44f-c3b07b36fab9-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.656452 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.656434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678cd7d5-7144-400c-a44f-c3b07b36fab9-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.656635 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.656475 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678cd7d5-7144-400c-a44f-c3b07b36fab9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.657242 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.657150 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678cd7d5-7144-400c-a44f-c3b07b36fab9-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.657361 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.657320 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678cd7d5-7144-400c-a44f-c3b07b36fab9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.657450 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.657428 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678cd7d5-7144-400c-a44f-c3b07b36fab9-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.658192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.658142 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678cd7d5-7144-400c-a44f-c3b07b36fab9-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.658564 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.658545 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678cd7d5-7144-400c-a44f-c3b07b36fab9-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.662085 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.662063 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678cd7d5-7144-400c-a44f-c3b07b36fab9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.662659 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.662636 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678cd7d5-7144-400c-a44f-c3b07b36fab9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.666101 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.666077 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/678cd7d5-7144-400c-a44f-c3b07b36fab9-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.666961 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.666927 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9fvh\" (UniqueName: \"kubernetes.io/projected/678cd7d5-7144-400c-a44f-c3b07b36fab9-kube-api-access-j9fvh\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tbk4x\" (UID: \"678cd7d5-7144-400c-a44f-c3b07b36fab9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.729948 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.729912 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:53.827752 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.827721 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:53.921479 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:53.921378 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x"] Jul 12 18:25:53.923965 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:25:53.923939 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678cd7d5_7144_400c_a44f_c3b07b36fab9.slice/crio-8035f0f23b927463771e7792387c8b6f5215140f95a7f85aeddd31c25ef548b8 WatchSource:0}: Error finding container 8035f0f23b927463771e7792387c8b6f5215140f95a7f85aeddd31c25ef548b8: Status 404 returned error can't find the container with id 8035f0f23b927463771e7792387c8b6f5215140f95a7f85aeddd31c25ef548b8 Jul 12 18:25:54.092437 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:54.092399 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" event={"ID":"678cd7d5-7144-400c-a44f-c3b07b36fab9","Type":"ContainerStarted","Data":"8035f0f23b927463771e7792387c8b6f5215140f95a7f85aeddd31c25ef548b8"} Jul 12 18:25:54.986633 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:54.986578 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:56.479449 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:56.479415 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:56.580116 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:56.580078 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:56.723756 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:56.723720 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:25:56.723855 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:56.723809 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:25:56.723895 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:56.723852 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:25:57.110047 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:57.110009 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" event={"ID":"678cd7d5-7144-400c-a44f-c3b07b36fab9","Type":"ContainerStarted","Data":"19f15beed2837f07ac2d5b08364eb45fe084c8aacf31de941c778b93e539d002"} Jul 12 18:25:57.130866 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:57.130814 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" podStartSLOduration=1.33381874 podStartE2EDuration="4.130780808s" totalImagesPullingTime="2.796962068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:25:57.128838922 +0000 UTC m=+518.896136504" watchObservedRunningTime="2026-07-12 18:25:57.130780808 +0000 UTC m=+518.898078389" Jul 12 18:25:57.730512 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:57.730471 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:57.735306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:57.735283 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:58.116368 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.116332 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:58.117346 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.117327 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:25:58.123269 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.123244 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:58.128831 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.128811 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zzl96"] Jul 12 18:25:58.132479 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.132464 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" Jul 12 18:25:58.134919 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.134899 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 12 18:25:58.143936 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.143903 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zzl96"] Jul 12 18:25:58.201341 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.201302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/28b8a786-a93b-4fa5-a8c3-6e1522d91eee-config-file\") pod \"limitador-limitador-64c94fd64f-zzl96\" (UID: \"28b8a786-a93b-4fa5-a8c3-6e1522d91eee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" Jul 12 18:25:58.201518 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.201359 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7tw8\" (UniqueName: \"kubernetes.io/projected/28b8a786-a93b-4fa5-a8c3-6e1522d91eee-kube-api-access-b7tw8\") pod \"limitador-limitador-64c94fd64f-zzl96\" (UID: \"28b8a786-a93b-4fa5-a8c3-6e1522d91eee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" Jul 12 18:25:58.226709 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.226677 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zzl96"] Jul 12 18:25:58.263317 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.263281 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:58.302257 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.302225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/28b8a786-a93b-4fa5-a8c3-6e1522d91eee-config-file\") pod \"limitador-limitador-64c94fd64f-zzl96\" (UID: \"28b8a786-a93b-4fa5-a8c3-6e1522d91eee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" Jul 12 18:25:58.302448 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.302265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7tw8\" (UniqueName: \"kubernetes.io/projected/28b8a786-a93b-4fa5-a8c3-6e1522d91eee-kube-api-access-b7tw8\") pod \"limitador-limitador-64c94fd64f-zzl96\" (UID: \"28b8a786-a93b-4fa5-a8c3-6e1522d91eee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" Jul 12 18:25:58.302944 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.302921 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/28b8a786-a93b-4fa5-a8c3-6e1522d91eee-config-file\") pod \"limitador-limitador-64c94fd64f-zzl96\" (UID: \"28b8a786-a93b-4fa5-a8c3-6e1522d91eee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" Jul 12 18:25:58.310866 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.310842 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7tw8\" (UniqueName: \"kubernetes.io/projected/28b8a786-a93b-4fa5-a8c3-6e1522d91eee-kube-api-access-b7tw8\") pod \"limitador-limitador-64c94fd64f-zzl96\" (UID: \"28b8a786-a93b-4fa5-a8c3-6e1522d91eee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" Jul 12 18:25:58.350645 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.350610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:58.444471 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.444386 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" Jul 12 18:25:58.582138 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.582116 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zzl96"] Jul 12 18:25:58.584244 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:25:58.584215 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b8a786_a93b_4fa5_a8c3_6e1522d91eee.slice/crio-953c2b808060f9615950a13b5adbc2fcd7fac1975318d64e65ea0c77205a354a WatchSource:0}: Error finding container 953c2b808060f9615950a13b5adbc2fcd7fac1975318d64e65ea0c77205a354a: Status 404 returned error can't find the container with id 953c2b808060f9615950a13b5adbc2fcd7fac1975318d64e65ea0c77205a354a Jul 12 18:25:58.937409 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:58.937371 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:59.035337 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:59.035306 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:25:59.040549 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:59.040515 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-xzzvc"] Jul 12 18:25:59.047159 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:59.047133 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xzzvc" Jul 12 18:25:59.047828 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:59.047801 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xzzvc"] Jul 12 18:25:59.049511 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:59.049492 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6gsx6\"" Jul 12 18:25:59.109280 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:59.109246 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsnhb\" (UniqueName: \"kubernetes.io/projected/b653ec88-3769-455a-a2b7-66a94372b0b1-kube-api-access-rsnhb\") pod \"authorino-75d556bd77-xzzvc\" (UID: \"b653ec88-3769-455a-a2b7-66a94372b0b1\") " pod="kuadrant-system/authorino-75d556bd77-xzzvc" Jul 12 18:25:59.123156 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:59.123118 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" event={"ID":"28b8a786-a93b-4fa5-a8c3-6e1522d91eee","Type":"ContainerStarted","Data":"953c2b808060f9615950a13b5adbc2fcd7fac1975318d64e65ea0c77205a354a"} Jul 12 18:25:59.211476 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:59.210733 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsnhb\" (UniqueName: \"kubernetes.io/projected/b653ec88-3769-455a-a2b7-66a94372b0b1-kube-api-access-rsnhb\") pod \"authorino-75d556bd77-xzzvc\" (UID: \"b653ec88-3769-455a-a2b7-66a94372b0b1\") " pod="kuadrant-system/authorino-75d556bd77-xzzvc" Jul 12 18:25:59.219566 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:59.219534 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsnhb\" (UniqueName: \"kubernetes.io/projected/b653ec88-3769-455a-a2b7-66a94372b0b1-kube-api-access-rsnhb\") pod \"authorino-75d556bd77-xzzvc\" (UID: \"b653ec88-3769-455a-a2b7-66a94372b0b1\") " pod="kuadrant-system/authorino-75d556bd77-xzzvc" Jul 12 18:25:59.359768 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:59.359492 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xzzvc" Jul 12 18:25:59.547687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:25:59.547540 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xzzvc"] Jul 12 18:25:59.550810 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:25:59.550762 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb653ec88_3769_455a_a2b7_66a94372b0b1.slice/crio-6bce3e9d590d54af36d54d61c089ea07f4ea051e74a0a2d2beb2eb2a8c34ef05 WatchSource:0}: Error finding container 6bce3e9d590d54af36d54d61c089ea07f4ea051e74a0a2d2beb2eb2a8c34ef05: Status 404 returned error can't find the container with id 6bce3e9d590d54af36d54d61c089ea07f4ea051e74a0a2d2beb2eb2a8c34ef05 Jul 12 18:26:00.136219 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:00.136161 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xzzvc" event={"ID":"b653ec88-3769-455a-a2b7-66a94372b0b1","Type":"ContainerStarted","Data":"6bce3e9d590d54af36d54d61c089ea07f4ea051e74a0a2d2beb2eb2a8c34ef05"} Jul 12 18:26:04.158449 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:04.158413 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" event={"ID":"28b8a786-a93b-4fa5-a8c3-6e1522d91eee","Type":"ContainerStarted","Data":"b6797539be80d82bfbd7c37ebc02388cdadc929926b5eb504b47d6eeef752eb4"} Jul 12 18:26:04.158936 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:04.158541 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" Jul 12 18:26:04.159863 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:04.159832 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xzzvc" event={"ID":"b653ec88-3769-455a-a2b7-66a94372b0b1","Type":"ContainerStarted","Data":"e0ac26eb7fbb01757cb2c91e21c24e27cd0a24e508b9f4ea9440b1c8fe9a7352"} Jul 12 18:26:04.174478 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:04.174385 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" podStartSLOduration=1.5799176190000002 podStartE2EDuration="6.174369704s" totalImagesPullingTime="4.594452085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:26:04.173663515 +0000 UTC m=+525.940961097" watchObservedRunningTime="2026-07-12 18:26:04.174369704 +0000 UTC m=+525.941667287" Jul 12 18:26:04.188146 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:04.188104 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-xzzvc" podStartSLOduration=1.552168751 podStartE2EDuration="5.188090372s" totalImagesPullingTime="3.635921621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:26:04.187371666 +0000 UTC m=+525.954669249" watchObservedRunningTime="2026-07-12 18:26:04.188090372 +0000 UTC m=+525.955387954" Jul 12 18:26:12.662024 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:12.661944 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zzl96"] Jul 12 18:26:12.662406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:12.662176 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" podUID="28b8a786-a93b-4fa5-a8c3-6e1522d91eee" containerName="limitador" containerID="cri-o://b6797539be80d82bfbd7c37ebc02388cdadc929926b5eb504b47d6eeef752eb4" gracePeriod=30 Jul 12 18:26:12.662866 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:12.662774 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" Jul 12 18:26:13.204038 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:13.204005 2576 generic.go:356] "Generic (PLEG): container finished" podID="28b8a786-a93b-4fa5-a8c3-6e1522d91eee" containerID="b6797539be80d82bfbd7c37ebc02388cdadc929926b5eb504b47d6eeef752eb4" exitCode=0 Jul 12 18:26:13.204038 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:13.204042 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" event={"ID":"28b8a786-a93b-4fa5-a8c3-6e1522d91eee","Type":"ContainerDied","Data":"b6797539be80d82bfbd7c37ebc02388cdadc929926b5eb504b47d6eeef752eb4"} Jul 12 18:26:13.604816 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:13.604771 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" Jul 12 18:26:13.746309 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:13.746227 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/28b8a786-a93b-4fa5-a8c3-6e1522d91eee-config-file\") pod \"28b8a786-a93b-4fa5-a8c3-6e1522d91eee\" (UID: \"28b8a786-a93b-4fa5-a8c3-6e1522d91eee\") " Jul 12 18:26:13.746309 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:13.746263 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7tw8\" (UniqueName: \"kubernetes.io/projected/28b8a786-a93b-4fa5-a8c3-6e1522d91eee-kube-api-access-b7tw8\") pod \"28b8a786-a93b-4fa5-a8c3-6e1522d91eee\" (UID: \"28b8a786-a93b-4fa5-a8c3-6e1522d91eee\") " Jul 12 18:26:13.746833 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:13.746594 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b8a786-a93b-4fa5-a8c3-6e1522d91eee-config-file" (OuterVolumeSpecName: "config-file") pod "28b8a786-a93b-4fa5-a8c3-6e1522d91eee" (UID: "28b8a786-a93b-4fa5-a8c3-6e1522d91eee"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:26:13.748327 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:13.748299 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b8a786-a93b-4fa5-a8c3-6e1522d91eee-kube-api-access-b7tw8" (OuterVolumeSpecName: "kube-api-access-b7tw8") pod "28b8a786-a93b-4fa5-a8c3-6e1522d91eee" (UID: "28b8a786-a93b-4fa5-a8c3-6e1522d91eee"). InnerVolumeSpecName "kube-api-access-b7tw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:26:13.847248 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:13.847214 2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/28b8a786-a93b-4fa5-a8c3-6e1522d91eee-config-file\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:26:13.847248 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:13.847242 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7tw8\" (UniqueName: \"kubernetes.io/projected/28b8a786-a93b-4fa5-a8c3-6e1522d91eee-kube-api-access-b7tw8\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:26:14.210005 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.209978 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" Jul 12 18:26:14.210179 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.209975 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zzl96" event={"ID":"28b8a786-a93b-4fa5-a8c3-6e1522d91eee","Type":"ContainerDied","Data":"953c2b808060f9615950a13b5adbc2fcd7fac1975318d64e65ea0c77205a354a"} Jul 12 18:26:14.210179 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.210112 2576 scope.go:117] "RemoveContainer" containerID="b6797539be80d82bfbd7c37ebc02388cdadc929926b5eb504b47d6eeef752eb4" Jul 12 18:26:14.229974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.229945 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zzl96"] Jul 12 18:26:14.233947 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.233927 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zzl96"] Jul 12 18:26:14.302441 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.302396 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:26:14.402595 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.402555 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:26:14.756647 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.756606 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vpwtc"] Jul 12 18:26:14.757218 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.757199 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28b8a786-a93b-4fa5-a8c3-6e1522d91eee" containerName="limitador" Jul 12 18:26:14.757299 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.757221 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="28b8a786-a93b-4fa5-a8c3-6e1522d91eee" containerName="limitador" Jul 12 18:26:14.757357 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.757338 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="28b8a786-a93b-4fa5-a8c3-6e1522d91eee" containerName="limitador" Jul 12 18:26:14.762004 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.761987 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vpwtc" Jul 12 18:26:14.764683 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.764652 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 12 18:26:14.764799 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.764740 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 12 18:26:14.764868 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.764772 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-4xc5x\"" Jul 12 18:26:14.765907 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.765881 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 12 18:26:14.770168 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.770130 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vpwtc"] Jul 12 18:26:14.791148 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.791120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:26:14.818194 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.818151 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28b8a786-a93b-4fa5-a8c3-6e1522d91eee" path="/var/lib/kubelet/pods/28b8a786-a93b-4fa5-a8c3-6e1522d91eee/volumes" Jul 12 18:26:14.957663 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.957627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8603f588-2204-4f00-86ba-94d7c69632a0-data\") pod \"postgres-69fb4594cf-vpwtc\" (UID: \"8603f588-2204-4f00-86ba-94d7c69632a0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vpwtc" Jul 12 18:26:14.957870 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:14.957675 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9z2c\" (UniqueName: \"kubernetes.io/projected/8603f588-2204-4f00-86ba-94d7c69632a0-kube-api-access-n9z2c\") pod \"postgres-69fb4594cf-vpwtc\" (UID: \"8603f588-2204-4f00-86ba-94d7c69632a0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vpwtc" Jul 12 18:26:15.058492 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:15.058408 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8603f588-2204-4f00-86ba-94d7c69632a0-data\") pod \"postgres-69fb4594cf-vpwtc\" (UID: \"8603f588-2204-4f00-86ba-94d7c69632a0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vpwtc" Jul 12 18:26:15.058492 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:15.058448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9z2c\" (UniqueName: \"kubernetes.io/projected/8603f588-2204-4f00-86ba-94d7c69632a0-kube-api-access-n9z2c\") pod \"postgres-69fb4594cf-vpwtc\" (UID: \"8603f588-2204-4f00-86ba-94d7c69632a0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vpwtc" Jul 12 18:26:15.058833 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:15.058812 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8603f588-2204-4f00-86ba-94d7c69632a0-data\") pod \"postgres-69fb4594cf-vpwtc\" (UID: \"8603f588-2204-4f00-86ba-94d7c69632a0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vpwtc" Jul 12 18:26:15.066751 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:15.066725 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9z2c\" (UniqueName: \"kubernetes.io/projected/8603f588-2204-4f00-86ba-94d7c69632a0-kube-api-access-n9z2c\") pod \"postgres-69fb4594cf-vpwtc\" (UID: \"8603f588-2204-4f00-86ba-94d7c69632a0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vpwtc" Jul 12 18:26:15.077576 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:15.077555 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vpwtc" Jul 12 18:26:15.207012 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:15.206978 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vpwtc"] Jul 12 18:26:15.208729 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:26:15.208705 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8603f588_2204_4f00_86ba_94d7c69632a0.slice/crio-5e64730c67efae3194194296ff8b2cdf7f4330de148de6832855fd98dc9d8c50 WatchSource:0}: Error finding container 5e64730c67efae3194194296ff8b2cdf7f4330de148de6832855fd98dc9d8c50: Status 404 returned error can't find the container with id 5e64730c67efae3194194296ff8b2cdf7f4330de148de6832855fd98dc9d8c50 Jul 12 18:26:15.216840 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:15.216816 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vpwtc" event={"ID":"8603f588-2204-4f00-86ba-94d7c69632a0","Type":"ContainerStarted","Data":"5e64730c67efae3194194296ff8b2cdf7f4330de148de6832855fd98dc9d8c50"} Jul 12 18:26:19.906058 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:19.906030 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 12 18:26:20.249306 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:20.249214 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vpwtc" event={"ID":"8603f588-2204-4f00-86ba-94d7c69632a0","Type":"ContainerStarted","Data":"0fa988390aa7d316da02251eefeb6cd2b9b394ce156ec9c0682e0f49c0e68ac6"} Jul 12 18:26:20.249472 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:20.249342 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vpwtc" Jul 12 18:26:20.265649 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:20.265598 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vpwtc" podStartSLOduration=1.572686967 podStartE2EDuration="6.265584663s" totalImagesPullingTime="4.692897696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:26:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:26:20.263835708 +0000 UTC m=+542.031133292" watchObservedRunningTime="2026-07-12 18:26:20.265584663 +0000 UTC m=+542.032882245" Jul 12 18:26:26.285067 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:26.285036 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vpwtc" Jul 12 18:26:26.598605 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:26.598574 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:26:26.843575 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:26.843541 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:26:28.820071 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:28.820040 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xzzvc"] Jul 12 18:26:28.820565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:28.820224 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-xzzvc" podUID="b653ec88-3769-455a-a2b7-66a94372b0b1" containerName="authorino" containerID="cri-o://e0ac26eb7fbb01757cb2c91e21c24e27cd0a24e508b9f4ea9440b1c8fe9a7352" gracePeriod=30 Jul 12 18:26:29.094319 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:29.094294 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xzzvc" Jul 12 18:26:29.188535 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:29.188501 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsnhb\" (UniqueName: \"kubernetes.io/projected/b653ec88-3769-455a-a2b7-66a94372b0b1-kube-api-access-rsnhb\") pod \"b653ec88-3769-455a-a2b7-66a94372b0b1\" (UID: \"b653ec88-3769-455a-a2b7-66a94372b0b1\") " Jul 12 18:26:29.190578 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:29.190552 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b653ec88-3769-455a-a2b7-66a94372b0b1-kube-api-access-rsnhb" (OuterVolumeSpecName: "kube-api-access-rsnhb") pod "b653ec88-3769-455a-a2b7-66a94372b0b1" (UID: "b653ec88-3769-455a-a2b7-66a94372b0b1"). InnerVolumeSpecName "kube-api-access-rsnhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:26:29.289968 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:29.289932 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsnhb\" (UniqueName: \"kubernetes.io/projected/b653ec88-3769-455a-a2b7-66a94372b0b1-kube-api-access-rsnhb\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:26:29.293894 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:29.293866 2576 generic.go:356] "Generic (PLEG): container finished" podID="b653ec88-3769-455a-a2b7-66a94372b0b1" containerID="e0ac26eb7fbb01757cb2c91e21c24e27cd0a24e508b9f4ea9440b1c8fe9a7352" exitCode=0 Jul 12 18:26:29.294021 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:29.293912 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xzzvc" Jul 12 18:26:29.294021 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:29.293945 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xzzvc" event={"ID":"b653ec88-3769-455a-a2b7-66a94372b0b1","Type":"ContainerDied","Data":"e0ac26eb7fbb01757cb2c91e21c24e27cd0a24e508b9f4ea9440b1c8fe9a7352"} Jul 12 18:26:29.294021 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:29.293988 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xzzvc" event={"ID":"b653ec88-3769-455a-a2b7-66a94372b0b1","Type":"ContainerDied","Data":"6bce3e9d590d54af36d54d61c089ea07f4ea051e74a0a2d2beb2eb2a8c34ef05"} Jul 12 18:26:29.294021 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:29.294008 2576 scope.go:117] "RemoveContainer" containerID="e0ac26eb7fbb01757cb2c91e21c24e27cd0a24e508b9f4ea9440b1c8fe9a7352" Jul 12 18:26:29.302595 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:29.302578 2576 scope.go:117] "RemoveContainer" containerID="e0ac26eb7fbb01757cb2c91e21c24e27cd0a24e508b9f4ea9440b1c8fe9a7352" Jul 12 18:26:29.302860 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:26:29.302843 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ac26eb7fbb01757cb2c91e21c24e27cd0a24e508b9f4ea9440b1c8fe9a7352\": container with ID starting with e0ac26eb7fbb01757cb2c91e21c24e27cd0a24e508b9f4ea9440b1c8fe9a7352 not found: ID does not exist" containerID="e0ac26eb7fbb01757cb2c91e21c24e27cd0a24e508b9f4ea9440b1c8fe9a7352" Jul 12 18:26:29.302919 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:29.302868 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ac26eb7fbb01757cb2c91e21c24e27cd0a24e508b9f4ea9440b1c8fe9a7352"} err="failed to get container status \"e0ac26eb7fbb01757cb2c91e21c24e27cd0a24e508b9f4ea9440b1c8fe9a7352\": rpc error: code = NotFound desc = could not find container \"e0ac26eb7fbb01757cb2c91e21c24e27cd0a24e508b9f4ea9440b1c8fe9a7352\": container with ID starting with e0ac26eb7fbb01757cb2c91e21c24e27cd0a24e508b9f4ea9440b1c8fe9a7352 not found: ID does not exist" Jul 12 18:26:29.314000 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:29.313974 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xzzvc"] Jul 12 18:26:29.317387 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:29.317365 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xzzvc"] Jul 12 18:26:30.818166 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:30.818123 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b653ec88-3769-455a-a2b7-66a94372b0b1" path="/var/lib/kubelet/pods/b653ec88-3769-455a-a2b7-66a94372b0b1/volumes" Jul 12 18:26:33.183698 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.183665 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5977755476-fhcvd"] Jul 12 18:26:33.184107 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.184070 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b653ec88-3769-455a-a2b7-66a94372b0b1" containerName="authorino" Jul 12 18:26:33.184107 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.184082 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b653ec88-3769-455a-a2b7-66a94372b0b1" containerName="authorino" Jul 12 18:26:33.184180 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.184150 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b653ec88-3769-455a-a2b7-66a94372b0b1" containerName="authorino" Jul 12 18:26:33.191403 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.191371 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:26:33.260889 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.260855 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5977755476-fhcvd"] Jul 12 18:26:33.261052 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.260954 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5977755476-fhcvd" Jul 12 18:26:33.265758 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.265728 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 12 18:26:33.265758 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.265754 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 12 18:26:33.265960 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.265818 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jrjhq\"" Jul 12 18:26:33.322239 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.322197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edc962b0-c1e7-43ba-81aa-0566fe08d970-webhook-cert\") pod \"maas-controller-5977755476-fhcvd\" (UID: \"edc962b0-c1e7-43ba-81aa-0566fe08d970\") " pod="opendatahub/maas-controller-5977755476-fhcvd" Jul 12 18:26:33.322239 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.322244 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w89wc\" (UniqueName: \"kubernetes.io/projected/edc962b0-c1e7-43ba-81aa-0566fe08d970-kube-api-access-w89wc\") pod \"maas-controller-5977755476-fhcvd\" (UID: \"edc962b0-c1e7-43ba-81aa-0566fe08d970\") " pod="opendatahub/maas-controller-5977755476-fhcvd" Jul 12 18:26:33.338750 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.338718 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:26:33.422883 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.422843 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edc962b0-c1e7-43ba-81aa-0566fe08d970-webhook-cert\") pod \"maas-controller-5977755476-fhcvd\" (UID: \"edc962b0-c1e7-43ba-81aa-0566fe08d970\") " pod="opendatahub/maas-controller-5977755476-fhcvd" Jul 12 18:26:33.423065 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.422891 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w89wc\" (UniqueName: \"kubernetes.io/projected/edc962b0-c1e7-43ba-81aa-0566fe08d970-kube-api-access-w89wc\") pod \"maas-controller-5977755476-fhcvd\" (UID: \"edc962b0-c1e7-43ba-81aa-0566fe08d970\") " pod="opendatahub/maas-controller-5977755476-fhcvd" Jul 12 18:26:33.423065 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:26:33.422991 2576 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 12 18:26:33.423161 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:26:33.423075 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc962b0-c1e7-43ba-81aa-0566fe08d970-webhook-cert podName:edc962b0-c1e7-43ba-81aa-0566fe08d970 nodeName:}" failed. No retries permitted until 2026-07-12 18:26:33.92305804 +0000 UTC m=+555.690355600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/edc962b0-c1e7-43ba-81aa-0566fe08d970-webhook-cert") pod "maas-controller-5977755476-fhcvd" (UID: "edc962b0-c1e7-43ba-81aa-0566fe08d970") : secret "maas-controller-webhook-cert" not found Jul 12 18:26:33.441729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.441654 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w89wc\" (UniqueName: \"kubernetes.io/projected/edc962b0-c1e7-43ba-81aa-0566fe08d970-kube-api-access-w89wc\") pod \"maas-controller-5977755476-fhcvd\" (UID: \"edc962b0-c1e7-43ba-81aa-0566fe08d970\") " pod="opendatahub/maas-controller-5977755476-fhcvd" Jul 12 18:26:33.486378 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.486348 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6bcb555768-kn555"] Jul 12 18:26:33.511254 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.511225 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bcb555768-kn555"] Jul 12 18:26:33.511394 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.511346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bcb555768-kn555" Jul 12 18:26:33.561511 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.561471 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:26:33.624690 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.624661 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17a161a7-e8f4-4521-9cbc-07e7608d7588-webhook-cert\") pod \"maas-controller-6bcb555768-kn555\" (UID: \"17a161a7-e8f4-4521-9cbc-07e7608d7588\") " pod="opendatahub/maas-controller-6bcb555768-kn555" Jul 12 18:26:33.624858 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.624715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7clp\" (UniqueName: \"kubernetes.io/projected/17a161a7-e8f4-4521-9cbc-07e7608d7588-kube-api-access-z7clp\") pod \"maas-controller-6bcb555768-kn555\" (UID: \"17a161a7-e8f4-4521-9cbc-07e7608d7588\") " pod="opendatahub/maas-controller-6bcb555768-kn555" Jul 12 18:26:33.641847 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.641817 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:26:33.725487 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.725402 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17a161a7-e8f4-4521-9cbc-07e7608d7588-webhook-cert\") pod \"maas-controller-6bcb555768-kn555\" (UID: \"17a161a7-e8f4-4521-9cbc-07e7608d7588\") " pod="opendatahub/maas-controller-6bcb555768-kn555" Jul 12 18:26:33.725487 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.725468 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7clp\" (UniqueName: \"kubernetes.io/projected/17a161a7-e8f4-4521-9cbc-07e7608d7588-kube-api-access-z7clp\") pod \"maas-controller-6bcb555768-kn555\" (UID: \"17a161a7-e8f4-4521-9cbc-07e7608d7588\") " pod="opendatahub/maas-controller-6bcb555768-kn555" Jul 12 18:26:33.727806 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.727770 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17a161a7-e8f4-4521-9cbc-07e7608d7588-webhook-cert\") pod \"maas-controller-6bcb555768-kn555\" (UID: \"17a161a7-e8f4-4521-9cbc-07e7608d7588\") " pod="opendatahub/maas-controller-6bcb555768-kn555" Jul 12 18:26:33.735077 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.735054 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7clp\" (UniqueName: \"kubernetes.io/projected/17a161a7-e8f4-4521-9cbc-07e7608d7588-kube-api-access-z7clp\") pod \"maas-controller-6bcb555768-kn555\" (UID: \"17a161a7-e8f4-4521-9cbc-07e7608d7588\") " pod="opendatahub/maas-controller-6bcb555768-kn555" Jul 12 18:26:33.822342 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.822307 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bcb555768-kn555" Jul 12 18:26:33.927410 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.927381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edc962b0-c1e7-43ba-81aa-0566fe08d970-webhook-cert\") pod \"maas-controller-5977755476-fhcvd\" (UID: \"edc962b0-c1e7-43ba-81aa-0566fe08d970\") " pod="opendatahub/maas-controller-5977755476-fhcvd" Jul 12 18:26:33.929927 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.929899 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edc962b0-c1e7-43ba-81aa-0566fe08d970-webhook-cert\") pod \"maas-controller-5977755476-fhcvd\" (UID: \"edc962b0-c1e7-43ba-81aa-0566fe08d970\") " pod="opendatahub/maas-controller-5977755476-fhcvd" Jul 12 18:26:33.972345 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:33.972317 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bcb555768-kn555"] Jul 12 18:26:33.975754 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:26:33.975698 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a161a7_e8f4_4521_9cbc_07e7608d7588.slice/crio-4c895b5da7e8c990fde73dea04f74cc5593c21eb78f21b6f34eb6f5521358152 WatchSource:0}: Error finding container 4c895b5da7e8c990fde73dea04f74cc5593c21eb78f21b6f34eb6f5521358152: Status 404 returned error can't find the container with id 4c895b5da7e8c990fde73dea04f74cc5593c21eb78f21b6f34eb6f5521358152 Jul 12 18:26:34.171260 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:34.171219 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5977755476-fhcvd" Jul 12 18:26:34.320366 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:34.320334 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bcb555768-kn555" event={"ID":"17a161a7-e8f4-4521-9cbc-07e7608d7588","Type":"ContainerStarted","Data":"4c895b5da7e8c990fde73dea04f74cc5593c21eb78f21b6f34eb6f5521358152"} Jul 12 18:26:34.342690 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:34.342665 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5977755476-fhcvd"] Jul 12 18:26:35.330157 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:35.330076 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5977755476-fhcvd" event={"ID":"edc962b0-c1e7-43ba-81aa-0566fe08d970","Type":"ContainerStarted","Data":"be86f038f06ccb0d7f8080f5cabf37a91ba4b69a78549d8b26a2960e9eb94c05"} Jul 12 18:26:36.894362 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:36.894283 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:26:37.091253 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:37.091222 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:26:37.341240 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:37.341201 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5977755476-fhcvd" event={"ID":"edc962b0-c1e7-43ba-81aa-0566fe08d970","Type":"ContainerStarted","Data":"eb1ae1e03e1fd748b0301786416a1df893ccaada46b5d38992cf84399b6bdf93"} Jul 12 18:26:37.341405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:37.341267 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5977755476-fhcvd" Jul 12 18:26:37.342566 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:37.342544 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bcb555768-kn555" event={"ID":"17a161a7-e8f4-4521-9cbc-07e7608d7588","Type":"ContainerStarted","Data":"891f64cb0bc172d11003e3d7f8d4467f315e47608d2931ea384a2a3a7e5ee50e"} Jul 12 18:26:37.342683 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:37.342664 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6bcb555768-kn555" Jul 12 18:26:37.375227 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:37.375177 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5977755476-fhcvd" podStartSLOduration=2.133155101 podStartE2EDuration="4.375161296s" totalImagesPullingTime="2.242006195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:26:37.372713025 +0000 UTC m=+559.140010607" watchObservedRunningTime="2026-07-12 18:26:37.375161296 +0000 UTC m=+559.142458877" Jul 12 18:26:37.424929 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:37.424877 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6bcb555768-kn555" podStartSLOduration=1.81172654 podStartE2EDuration="4.424860722s" totalImagesPullingTime="2.613134182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:26:37.424253031 +0000 UTC m=+559.191550607" watchObservedRunningTime="2026-07-12 18:26:37.424860722 +0000 UTC m=+559.192158349" Jul 12 18:26:48.355480 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:48.355449 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6bcb555768-kn555" Jul 12 18:26:48.355909 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:48.355525 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5977755476-fhcvd" Jul 12 18:26:48.414089 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:48.414055 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5977755476-fhcvd"] Jul 12 18:26:48.414289 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:48.414251 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5977755476-fhcvd" podUID="edc962b0-c1e7-43ba-81aa-0566fe08d970" containerName="manager" containerID="cri-o://eb1ae1e03e1fd748b0301786416a1df893ccaada46b5d38992cf84399b6bdf93" gracePeriod=30 Jul 12 18:26:48.661871 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:48.661847 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5977755476-fhcvd" Jul 12 18:26:48.759359 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:48.759326 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w89wc\" (UniqueName: \"kubernetes.io/projected/edc962b0-c1e7-43ba-81aa-0566fe08d970-kube-api-access-w89wc\") pod \"edc962b0-c1e7-43ba-81aa-0566fe08d970\" (UID: \"edc962b0-c1e7-43ba-81aa-0566fe08d970\") " Jul 12 18:26:48.759521 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:48.759400 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edc962b0-c1e7-43ba-81aa-0566fe08d970-webhook-cert\") pod \"edc962b0-c1e7-43ba-81aa-0566fe08d970\" (UID: \"edc962b0-c1e7-43ba-81aa-0566fe08d970\") " Jul 12 18:26:48.761624 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:48.761591 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc962b0-c1e7-43ba-81aa-0566fe08d970-kube-api-access-w89wc" (OuterVolumeSpecName: "kube-api-access-w89wc") pod "edc962b0-c1e7-43ba-81aa-0566fe08d970" (UID: "edc962b0-c1e7-43ba-81aa-0566fe08d970"). InnerVolumeSpecName "kube-api-access-w89wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:26:48.761624 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:48.761609 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edc962b0-c1e7-43ba-81aa-0566fe08d970-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "edc962b0-c1e7-43ba-81aa-0566fe08d970" (UID: "edc962b0-c1e7-43ba-81aa-0566fe08d970"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:26:48.860669 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:48.860644 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w89wc\" (UniqueName: \"kubernetes.io/projected/edc962b0-c1e7-43ba-81aa-0566fe08d970-kube-api-access-w89wc\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:26:48.860669 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:48.860669 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edc962b0-c1e7-43ba-81aa-0566fe08d970-webhook-cert\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:26:49.408657 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:49.408621 2576 generic.go:356] "Generic (PLEG): container finished" podID="edc962b0-c1e7-43ba-81aa-0566fe08d970" containerID="eb1ae1e03e1fd748b0301786416a1df893ccaada46b5d38992cf84399b6bdf93" exitCode=0 Jul 12 18:26:49.409111 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:49.408670 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5977755476-fhcvd" event={"ID":"edc962b0-c1e7-43ba-81aa-0566fe08d970","Type":"ContainerDied","Data":"eb1ae1e03e1fd748b0301786416a1df893ccaada46b5d38992cf84399b6bdf93"} Jul 12 18:26:49.409111 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:49.408710 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5977755476-fhcvd" event={"ID":"edc962b0-c1e7-43ba-81aa-0566fe08d970","Type":"ContainerDied","Data":"be86f038f06ccb0d7f8080f5cabf37a91ba4b69a78549d8b26a2960e9eb94c05"} Jul 12 18:26:49.409111 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:49.408732 2576 scope.go:117] "RemoveContainer" containerID="eb1ae1e03e1fd748b0301786416a1df893ccaada46b5d38992cf84399b6bdf93" Jul 12 18:26:49.409111 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:49.408673 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5977755476-fhcvd" Jul 12 18:26:49.417229 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:49.417052 2576 scope.go:117] "RemoveContainer" containerID="eb1ae1e03e1fd748b0301786416a1df893ccaada46b5d38992cf84399b6bdf93" Jul 12 18:26:49.417291 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:26:49.417272 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1ae1e03e1fd748b0301786416a1df893ccaada46b5d38992cf84399b6bdf93\": container with ID starting with eb1ae1e03e1fd748b0301786416a1df893ccaada46b5d38992cf84399b6bdf93 not found: ID does not exist" containerID="eb1ae1e03e1fd748b0301786416a1df893ccaada46b5d38992cf84399b6bdf93" Jul 12 18:26:49.417332 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:49.417299 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1ae1e03e1fd748b0301786416a1df893ccaada46b5d38992cf84399b6bdf93"} err="failed to get container status \"eb1ae1e03e1fd748b0301786416a1df893ccaada46b5d38992cf84399b6bdf93\": rpc error: code = NotFound desc = could not find container \"eb1ae1e03e1fd748b0301786416a1df893ccaada46b5d38992cf84399b6bdf93\": container with ID starting with eb1ae1e03e1fd748b0301786416a1df893ccaada46b5d38992cf84399b6bdf93 not found: ID does not exist" Jul 12 18:26:49.427313 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:49.427293 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5977755476-fhcvd"] Jul 12 18:26:49.431473 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:49.431443 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5977755476-fhcvd"] Jul 12 18:26:50.817559 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:50.817524 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edc962b0-c1e7-43ba-81aa-0566fe08d970" path="/var/lib/kubelet/pods/edc962b0-c1e7-43ba-81aa-0566fe08d970/volumes" Jul 12 18:26:55.121835 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:26:55.121779 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:27:09.463155 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:09.463117 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:27:10.028723 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.028686 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:27:10.077644 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.077615 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64"] Jul 12 18:27:10.078047 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.078025 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc962b0-c1e7-43ba-81aa-0566fe08d970" containerName="manager" Jul 12 18:27:10.078047 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.078040 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc962b0-c1e7-43ba-81aa-0566fe08d970" containerName="manager" Jul 12 18:27:10.078199 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.078110 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc962b0-c1e7-43ba-81aa-0566fe08d970" containerName="manager" Jul 12 18:27:10.101630 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.101600 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64"] Jul 12 18:27:10.101777 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.101691 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64" Jul 12 18:27:10.105243 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.105221 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pwj8p\"" Jul 12 18:27:10.106269 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.106254 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 12 18:27:10.152066 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.152027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b42f\" (UniqueName: \"kubernetes.io/projected/3507b0b7-e7ea-48ee-ae77-6594d7cb5f83-kube-api-access-2b42f\") pod \"maas-api-7c864c6f55-z9r64\" (UID: \"3507b0b7-e7ea-48ee-ae77-6594d7cb5f83\") " pod="odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64" Jul 12 18:27:10.152245 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.152074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3507b0b7-e7ea-48ee-ae77-6594d7cb5f83-openshift-service-ca\") pod \"maas-api-7c864c6f55-z9r64\" (UID: \"3507b0b7-e7ea-48ee-ae77-6594d7cb5f83\") " pod="odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64" Jul 12 18:27:10.152245 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.152197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3507b0b7-e7ea-48ee-ae77-6594d7cb5f83-maas-api-tls\") pod \"maas-api-7c864c6f55-z9r64\" (UID: \"3507b0b7-e7ea-48ee-ae77-6594d7cb5f83\") " pod="odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64" Jul 12 18:27:10.153147 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.153097 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:27:10.253360 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.253318 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3507b0b7-e7ea-48ee-ae77-6594d7cb5f83-openshift-service-ca\") pod \"maas-api-7c864c6f55-z9r64\" (UID: \"3507b0b7-e7ea-48ee-ae77-6594d7cb5f83\") " pod="odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64" Jul 12 18:27:10.253568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.253405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3507b0b7-e7ea-48ee-ae77-6594d7cb5f83-maas-api-tls\") pod \"maas-api-7c864c6f55-z9r64\" (UID: \"3507b0b7-e7ea-48ee-ae77-6594d7cb5f83\") " pod="odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64" Jul 12 18:27:10.253568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.253449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b42f\" (UniqueName: \"kubernetes.io/projected/3507b0b7-e7ea-48ee-ae77-6594d7cb5f83-kube-api-access-2b42f\") pod \"maas-api-7c864c6f55-z9r64\" (UID: \"3507b0b7-e7ea-48ee-ae77-6594d7cb5f83\") " pod="odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64" Jul 12 18:27:10.254076 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.254052 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3507b0b7-e7ea-48ee-ae77-6594d7cb5f83-openshift-service-ca\") pod \"maas-api-7c864c6f55-z9r64\" (UID: \"3507b0b7-e7ea-48ee-ae77-6594d7cb5f83\") " pod="odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64" Jul 12 18:27:10.255752 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.255730 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3507b0b7-e7ea-48ee-ae77-6594d7cb5f83-maas-api-tls\") pod \"maas-api-7c864c6f55-z9r64\" (UID: \"3507b0b7-e7ea-48ee-ae77-6594d7cb5f83\") " pod="odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64" Jul 12 18:27:10.273504 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.273476 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b42f\" (UniqueName: \"kubernetes.io/projected/3507b0b7-e7ea-48ee-ae77-6594d7cb5f83-kube-api-access-2b42f\") pod \"maas-api-7c864c6f55-z9r64\" (UID: \"3507b0b7-e7ea-48ee-ae77-6594d7cb5f83\") " pod="odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64" Jul 12 18:27:10.413487 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.413450 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64" Jul 12 18:27:10.563713 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.563689 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64"] Jul 12 18:27:10.566369 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:27:10.566340 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3507b0b7_e7ea_48ee_ae77_6594d7cb5f83.slice/crio-0623947b6d22a1429e61ff1b57792c7597b8f4749b2e32a395f694123a1dc796 WatchSource:0}: Error finding container 0623947b6d22a1429e61ff1b57792c7597b8f4749b2e32a395f694123a1dc796: Status 404 returned error can't find the container with id 0623947b6d22a1429e61ff1b57792c7597b8f4749b2e32a395f694123a1dc796 Jul 12 18:27:10.812266 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.812172 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:27:10.920630 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.920598 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:27:10.988650 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.988615 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-9msj9"] Jul 12 18:27:10.993588 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.993567 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-9msj9" Jul 12 18:27:10.996844 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.996824 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-z6g2m\"" Jul 12 18:27:10.998320 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:10.998297 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 12 18:27:11.017628 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.017599 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-9msj9"] Jul 12 18:27:11.058623 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.058595 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv5t8\" (UniqueName: \"kubernetes.io/projected/53951bfe-0dde-4ee0-9199-f2de0afb1c82-kube-api-access-nv5t8\") pod \"payload-pre-processing-7964457696-9msj9\" (UID: \"53951bfe-0dde-4ee0-9199-f2de0afb1c82\") " pod="openshift-ingress/payload-pre-processing-7964457696-9msj9" Jul 12 18:27:11.058815 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.058674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53951bfe-0dde-4ee0-9199-f2de0afb1c82-plugins-config-volume\") pod \"payload-pre-processing-7964457696-9msj9\" (UID: \"53951bfe-0dde-4ee0-9199-f2de0afb1c82\") " pod="openshift-ingress/payload-pre-processing-7964457696-9msj9" Jul 12 18:27:11.143625 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.143596 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-v9qgm"] Jul 12 18:27:11.147503 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.147483 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-v9qgm" Jul 12 18:27:11.160114 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.160070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv5t8\" (UniqueName: \"kubernetes.io/projected/53951bfe-0dde-4ee0-9199-f2de0afb1c82-kube-api-access-nv5t8\") pod \"payload-pre-processing-7964457696-9msj9\" (UID: \"53951bfe-0dde-4ee0-9199-f2de0afb1c82\") " pod="openshift-ingress/payload-pre-processing-7964457696-9msj9" Jul 12 18:27:11.160263 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.160119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629ea87f-db94-4c2c-ae5e-f1f48cb5d378-plugins-config-volume\") pod \"payload-processing-58f86c7798-v9qgm\" (UID: \"629ea87f-db94-4c2c-ae5e-f1f48cb5d378\") " pod="openshift-ingress/payload-processing-58f86c7798-v9qgm" Jul 12 18:27:11.160263 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.160197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfdjt\" (UniqueName: \"kubernetes.io/projected/629ea87f-db94-4c2c-ae5e-f1f48cb5d378-kube-api-access-vfdjt\") pod \"payload-processing-58f86c7798-v9qgm\" (UID: \"629ea87f-db94-4c2c-ae5e-f1f48cb5d378\") " pod="openshift-ingress/payload-processing-58f86c7798-v9qgm" Jul 12 18:27:11.160381 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.160273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53951bfe-0dde-4ee0-9199-f2de0afb1c82-plugins-config-volume\") pod \"payload-pre-processing-7964457696-9msj9\" (UID: \"53951bfe-0dde-4ee0-9199-f2de0afb1c82\") " pod="openshift-ingress/payload-pre-processing-7964457696-9msj9" Jul 12 18:27:11.161114 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.161086 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53951bfe-0dde-4ee0-9199-f2de0afb1c82-plugins-config-volume\") pod \"payload-pre-processing-7964457696-9msj9\" (UID: \"53951bfe-0dde-4ee0-9199-f2de0afb1c82\") " pod="openshift-ingress/payload-pre-processing-7964457696-9msj9" Jul 12 18:27:11.162312 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.162292 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-v9qgm"] Jul 12 18:27:11.185015 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.184988 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv5t8\" (UniqueName: \"kubernetes.io/projected/53951bfe-0dde-4ee0-9199-f2de0afb1c82-kube-api-access-nv5t8\") pod \"payload-pre-processing-7964457696-9msj9\" (UID: \"53951bfe-0dde-4ee0-9199-f2de0afb1c82\") " pod="openshift-ingress/payload-pre-processing-7964457696-9msj9" Jul 12 18:27:11.262503 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.262059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629ea87f-db94-4c2c-ae5e-f1f48cb5d378-plugins-config-volume\") pod \"payload-processing-58f86c7798-v9qgm\" (UID: \"629ea87f-db94-4c2c-ae5e-f1f48cb5d378\") " pod="openshift-ingress/payload-processing-58f86c7798-v9qgm" Jul 12 18:27:11.262503 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.262144 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfdjt\" (UniqueName: \"kubernetes.io/projected/629ea87f-db94-4c2c-ae5e-f1f48cb5d378-kube-api-access-vfdjt\") pod \"payload-processing-58f86c7798-v9qgm\" (UID: \"629ea87f-db94-4c2c-ae5e-f1f48cb5d378\") " pod="openshift-ingress/payload-processing-58f86c7798-v9qgm" Jul 12 18:27:11.263566 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.263540 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629ea87f-db94-4c2c-ae5e-f1f48cb5d378-plugins-config-volume\") pod \"payload-processing-58f86c7798-v9qgm\" (UID: \"629ea87f-db94-4c2c-ae5e-f1f48cb5d378\") " pod="openshift-ingress/payload-processing-58f86c7798-v9qgm" Jul 12 18:27:11.300572 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.300536 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfdjt\" (UniqueName: \"kubernetes.io/projected/629ea87f-db94-4c2c-ae5e-f1f48cb5d378-kube-api-access-vfdjt\") pod \"payload-processing-58f86c7798-v9qgm\" (UID: \"629ea87f-db94-4c2c-ae5e-f1f48cb5d378\") " pod="openshift-ingress/payload-processing-58f86c7798-v9qgm" Jul 12 18:27:11.305236 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.305183 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-9msj9" Jul 12 18:27:11.457281 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.457188 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-v9qgm" Jul 12 18:27:11.463956 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.463769 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-9msj9"] Jul 12 18:27:11.466993 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:27:11.466962 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53951bfe_0dde_4ee0_9199_f2de0afb1c82.slice/crio-0dd819dcc48ed66456161e068c6b8c61ba786ea22a2ddd2b17cb5a7124b9d66d WatchSource:0}: Error finding container 0dd819dcc48ed66456161e068c6b8c61ba786ea22a2ddd2b17cb5a7124b9d66d: Status 404 returned error can't find the container with id 0dd819dcc48ed66456161e068c6b8c61ba786ea22a2ddd2b17cb5a7124b9d66d Jul 12 18:27:11.520746 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.520708 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-9msj9" event={"ID":"53951bfe-0dde-4ee0-9199-f2de0afb1c82","Type":"ContainerStarted","Data":"0dd819dcc48ed66456161e068c6b8c61ba786ea22a2ddd2b17cb5a7124b9d66d"} Jul 12 18:27:11.523198 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.523120 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64" event={"ID":"3507b0b7-e7ea-48ee-ae77-6594d7cb5f83","Type":"ContainerStarted","Data":"0623947b6d22a1429e61ff1b57792c7597b8f4749b2e32a395f694123a1dc796"} Jul 12 18:27:11.635260 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:11.635234 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-v9qgm"] Jul 12 18:27:11.638225 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:27:11.638192 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629ea87f_db94_4c2c_ae5e_f1f48cb5d378.slice/crio-d5fca8913631cfd8e44c298411f6d1e826163653a95295f79ec4513b0b889d38 WatchSource:0}: Error finding container d5fca8913631cfd8e44c298411f6d1e826163653a95295f79ec4513b0b889d38: Status 404 returned error can't find the container with id d5fca8913631cfd8e44c298411f6d1e826163653a95295f79ec4513b0b889d38 Jul 12 18:27:12.551427 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:12.551379 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-v9qgm" event={"ID":"629ea87f-db94-4c2c-ae5e-f1f48cb5d378","Type":"ContainerStarted","Data":"d5fca8913631cfd8e44c298411f6d1e826163653a95295f79ec4513b0b889d38"} Jul 12 18:27:13.558965 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:13.558926 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-9msj9" event={"ID":"53951bfe-0dde-4ee0-9199-f2de0afb1c82","Type":"ContainerStarted","Data":"4e85062630a5e514f0253792f9a2f8ea8df826734f6e5306a53b3e185f2ded22"} Jul 12 18:27:13.559426 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:13.558992 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-9msj9" Jul 12 18:27:13.560518 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:13.560489 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64" event={"ID":"3507b0b7-e7ea-48ee-ae77-6594d7cb5f83","Type":"ContainerStarted","Data":"6c9808f4cd1d1cf9a3808ec4e03c1f921a2d0e9067f6009eb2ca94db360b5592"} Jul 12 18:27:13.560642 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:13.560567 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64" Jul 12 18:27:13.561947 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:13.561924 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-v9qgm" event={"ID":"629ea87f-db94-4c2c-ae5e-f1f48cb5d378","Type":"ContainerStarted","Data":"1642ebfc71c3bd0a966f27eb04df7e555dd67a105c18e47c3419dbf68fd7e242"} Jul 12 18:27:13.562028 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:13.561976 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-v9qgm" Jul 12 18:27:13.584971 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:13.584930 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-9msj9" podStartSLOduration=1.95948261 podStartE2EDuration="3.584917141s" totalImagesPullingTime="1.625434531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:27:13.583300971 +0000 UTC m=+595.350598545" watchObservedRunningTime="2026-07-12 18:27:13.584917141 +0000 UTC m=+595.352214722" Jul 12 18:27:13.604642 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:13.604599 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-v9qgm" podStartSLOduration=1.151921899 podStartE2EDuration="2.604585547s" totalImagesPullingTime="1.452663648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:27:13.603772856 +0000 UTC m=+595.371070437" watchObservedRunningTime="2026-07-12 18:27:13.604585547 +0000 UTC m=+595.371883128" Jul 12 18:27:13.624651 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:13.624604 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64" podStartSLOduration=1.185872399 podStartE2EDuration="3.624590048s" totalImagesPullingTime="2.438717649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:27:13.624208102 +0000 UTC m=+595.391505681" watchObservedRunningTime="2026-07-12 18:27:13.624590048 +0000 UTC m=+595.391887629" Jul 12 18:27:19.572543 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:19.572506 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c864c6f55-z9r64" Jul 12 18:27:19.592037 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:19.592006 2576 generic.go:356] "Generic (PLEG): container finished" podID="678cd7d5-7144-400c-a44f-c3b07b36fab9" containerID="19f15beed2837f07ac2d5b08364eb45fe084c8aacf31de941c778b93e539d002" exitCode=0 Jul 12 18:27:19.592183 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:19.592081 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" event={"ID":"678cd7d5-7144-400c-a44f-c3b07b36fab9","Type":"ContainerDied","Data":"19f15beed2837f07ac2d5b08364eb45fe084c8aacf31de941c778b93e539d002"} Jul 12 18:27:19.592471 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:19.592455 2576 scope.go:117] "RemoveContainer" containerID="19f15beed2837f07ac2d5b08364eb45fe084c8aacf31de941c778b93e539d002" Jul 12 18:27:19.593414 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:19.593385 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:27:19.593488 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:19.593453 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:27:19.593526 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:19.593515 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:27:20.599288 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:20.599250 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" event={"ID":"678cd7d5-7144-400c-a44f-c3b07b36fab9","Type":"ContainerStarted","Data":"830607d55824ed35dc835b849871774569d3aa5202aeb9385137eb1b9f6c692b"} Jul 12 18:27:20.657393 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:20.657355 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:27:20.730452 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:20.730407 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:27:20.730649 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:20.730474 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-tbk4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 12 18:27:20.730649 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:20.730520 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" podUID="678cd7d5-7144-400c-a44f-c3b07b36fab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 12 18:27:20.859919 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:20.859826 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:27:21.730621 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:21.730577 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-tbk4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 12 18:27:21.731119 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:21.730646 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" podUID="678cd7d5-7144-400c-a44f-c3b07b36fab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 12 18:27:22.731391 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:22.731353 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-tbk4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 12 18:27:22.731906 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:22.731425 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" podUID="678cd7d5-7144-400c-a44f-c3b07b36fab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 12 18:27:23.730729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:23.730693 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-tbk4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 12 18:27:23.731048 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:23.731020 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" podUID="678cd7d5-7144-400c-a44f-c3b07b36fab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 12 18:27:24.569403 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:24.569372 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-v9qgm" Jul 12 18:27:24.569838 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:24.569451 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-9msj9" Jul 12 18:27:24.735118 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:24.735091 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:27:24.735480 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:24.735457 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:27:24.736058 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:24.736037 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tbk4x" Jul 12 18:27:28.305273 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.305238 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb"] Jul 12 18:27:28.309896 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.309875 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.313823 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.313777 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 12 18:27:28.313947 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.313826 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 12 18:27:28.313947 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.313859 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 12 18:27:28.313947 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.313873 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zf69s\"" Jul 12 18:27:28.321536 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.321512 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb"] Jul 12 18:27:28.323723 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.323697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:27:28.423406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.423370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f4958c-af58-421a-bcd7-1396d6f010d6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.423406 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.423408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f4958c-af58-421a-bcd7-1396d6f010d6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.423625 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.423455 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f4958c-af58-421a-bcd7-1396d6f010d6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.423625 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.423475 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f4958c-af58-421a-bcd7-1396d6f010d6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.423625 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.423504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f4958c-af58-421a-bcd7-1396d6f010d6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.423625 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.423553 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnn47\" (UniqueName: \"kubernetes.io/projected/04f4958c-af58-421a-bcd7-1396d6f010d6-kube-api-access-pnn47\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.524081 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.524049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f4958c-af58-421a-bcd7-1396d6f010d6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.524081 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.524087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f4958c-af58-421a-bcd7-1396d6f010d6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.524296 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.524140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f4958c-af58-421a-bcd7-1396d6f010d6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.524296 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.524164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f4958c-af58-421a-bcd7-1396d6f010d6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.524296 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.524205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f4958c-af58-421a-bcd7-1396d6f010d6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.524296 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.524237 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnn47\" (UniqueName: \"kubernetes.io/projected/04f4958c-af58-421a-bcd7-1396d6f010d6-kube-api-access-pnn47\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.524508 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.524468 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f4958c-af58-421a-bcd7-1396d6f010d6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.524566 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.524525 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f4958c-af58-421a-bcd7-1396d6f010d6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.524622 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.524592 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f4958c-af58-421a-bcd7-1396d6f010d6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.526654 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.526628 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f4958c-af58-421a-bcd7-1396d6f010d6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.526822 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.526704 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f4958c-af58-421a-bcd7-1396d6f010d6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.533028 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.532995 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnn47\" (UniqueName: \"kubernetes.io/projected/04f4958c-af58-421a-bcd7-1396d6f010d6-kube-api-access-pnn47\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb\" (UID: \"04f4958c-af58-421a-bcd7-1396d6f010d6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.620974 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.620935 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:28.775639 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:28.775615 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb"] Jul 12 18:27:28.778202 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:27:28.778171 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f4958c_af58_421a_bcd7_1396d6f010d6.slice/crio-42e54fb335e535174d22ebec7424ca240b5085e98722b4a2d5bd71d412742f8f WatchSource:0}: Error finding container 42e54fb335e535174d22ebec7424ca240b5085e98722b4a2d5bd71d412742f8f: Status 404 returned error can't find the container with id 42e54fb335e535174d22ebec7424ca240b5085e98722b4a2d5bd71d412742f8f Jul 12 18:27:29.645480 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:29.645445 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" event={"ID":"04f4958c-af58-421a-bcd7-1396d6f010d6","Type":"ContainerStarted","Data":"42e54fb335e535174d22ebec7424ca240b5085e98722b4a2d5bd71d412742f8f"} Jul 12 18:27:30.651300 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:30.651263 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" event={"ID":"04f4958c-af58-421a-bcd7-1396d6f010d6","Type":"ContainerStarted","Data":"19fb01256079496c6822995c10320f8fa2a475325dd8936525fde3c2bb59edfa"} Jul 12 18:27:30.685164 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:30.685113 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" podStartSLOduration=1.5562617319999998 podStartE2EDuration="2.685099129s" totalImagesPullingTime="1.128837397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:27:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:27:30.683059876 +0000 UTC m=+612.450357459" watchObservedRunningTime="2026-07-12 18:27:30.685099129 +0000 UTC m=+612.452396710" Jul 12 18:27:38.622074 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:38.622027 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:38.622520 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:38.622097 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:38.634529 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:38.634500 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:38.702734 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:38.702708 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb" Jul 12 18:27:39.991575 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:39.991537 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll"] Jul 12 18:27:39.995947 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:39.995921 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.000667 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.000644 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 12 18:27:40.011888 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.011867 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll"] Jul 12 18:27:40.083226 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.083199 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:27:40.126088 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.125995 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c75a7175-8860-414c-980d-b22d57f898cd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.126088 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.126060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnpt4\" (UniqueName: \"kubernetes.io/projected/c75a7175-8860-414c-980d-b22d57f898cd-kube-api-access-vnpt4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.126304 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.126093 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c75a7175-8860-414c-980d-b22d57f898cd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.126304 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.126118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c75a7175-8860-414c-980d-b22d57f898cd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.126304 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.126222 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c75a7175-8860-414c-980d-b22d57f898cd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.126304 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.126284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c75a7175-8860-414c-980d-b22d57f898cd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.226910 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.226872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c75a7175-8860-414c-980d-b22d57f898cd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.227081 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.226921 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnpt4\" (UniqueName: \"kubernetes.io/projected/c75a7175-8860-414c-980d-b22d57f898cd-kube-api-access-vnpt4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.227081 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.226942 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c75a7175-8860-414c-980d-b22d57f898cd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.227081 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.226960 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c75a7175-8860-414c-980d-b22d57f898cd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.227081 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.226984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c75a7175-8860-414c-980d-b22d57f898cd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.227081 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.227018 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c75a7175-8860-414c-980d-b22d57f898cd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.227400 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.227376 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c75a7175-8860-414c-980d-b22d57f898cd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.227464 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.227449 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c75a7175-8860-414c-980d-b22d57f898cd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.227638 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.227610 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c75a7175-8860-414c-980d-b22d57f898cd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.229640 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.229608 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c75a7175-8860-414c-980d-b22d57f898cd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.229729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.229664 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c75a7175-8860-414c-980d-b22d57f898cd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.241873 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.241815 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnpt4\" (UniqueName: \"kubernetes.io/projected/c75a7175-8860-414c-980d-b22d57f898cd-kube-api-access-vnpt4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ztcll\" (UID: \"c75a7175-8860-414c-980d-b22d57f898cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.309635 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.309600 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:40.474706 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.474680 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll"] Jul 12 18:27:40.477336 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:27:40.477310 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75a7175_8860_414c_980d_b22d57f898cd.slice/crio-9601885352ea282878119677dd6432d69b7381b685762f6ff6adeeab02a96b5d WatchSource:0}: Error finding container 9601885352ea282878119677dd6432d69b7381b685762f6ff6adeeab02a96b5d: Status 404 returned error can't find the container with id 9601885352ea282878119677dd6432d69b7381b685762f6ff6adeeab02a96b5d Jul 12 18:27:40.702906 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.702876 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" event={"ID":"c75a7175-8860-414c-980d-b22d57f898cd","Type":"ContainerStarted","Data":"2a517af9302d93dce7040b7f447f451daec978347a0609edbf68ab5abd91756d"} Jul 12 18:27:40.703022 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:40.702933 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" event={"ID":"c75a7175-8860-414c-980d-b22d57f898cd","Type":"ContainerStarted","Data":"9601885352ea282878119677dd6432d69b7381b685762f6ff6adeeab02a96b5d"} Jul 12 18:27:50.310780 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:50.310739 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:50.311252 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:50.310840 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:50.323294 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:50.323261 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:50.355614 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:50.355558 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" podStartSLOduration=11.190079255 podStartE2EDuration="11.355515968s" totalImagesPullingTime="165.436713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:27:40.742587584 +0000 UTC m=+622.509885169" watchObservedRunningTime="2026-07-12 18:27:50.355515968 +0000 UTC m=+632.122813547" Jul 12 18:27:50.762839 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:50.762812 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ztcll" Jul 12 18:27:53.716807 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:27:53.716749 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:28:04.241867 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:28:04.241831 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:28:06.116673 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:28:06.116632 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:28:13.414697 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:28:13.414654 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:28:20.033849 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:28:20.033805 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:29:25.226188 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:25.226097 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7ccdcbfdd6-8lv2s"] Jul 12 18:29:25.230459 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:25.230432 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ccdcbfdd6-8lv2s" Jul 12 18:29:25.234312 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:25.234282 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6gsx6\"" Jul 12 18:29:25.234453 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:25.234286 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 12 18:29:25.236723 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:25.236700 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7ccdcbfdd6-8lv2s"] Jul 12 18:29:25.400808 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:25.400751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4d715528-b0d2-4eed-b118-396125ab82c7-tls-cert\") pod \"authorino-7ccdcbfdd6-8lv2s\" (UID: \"4d715528-b0d2-4eed-b118-396125ab82c7\") " pod="kuadrant-system/authorino-7ccdcbfdd6-8lv2s" Jul 12 18:29:25.400970 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:25.400862 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h49x2\" (UniqueName: \"kubernetes.io/projected/4d715528-b0d2-4eed-b118-396125ab82c7-kube-api-access-h49x2\") pod \"authorino-7ccdcbfdd6-8lv2s\" (UID: \"4d715528-b0d2-4eed-b118-396125ab82c7\") " pod="kuadrant-system/authorino-7ccdcbfdd6-8lv2s" Jul 12 18:29:25.502234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:25.502146 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h49x2\" (UniqueName: \"kubernetes.io/projected/4d715528-b0d2-4eed-b118-396125ab82c7-kube-api-access-h49x2\") pod \"authorino-7ccdcbfdd6-8lv2s\" (UID: \"4d715528-b0d2-4eed-b118-396125ab82c7\") " pod="kuadrant-system/authorino-7ccdcbfdd6-8lv2s" Jul 12 18:29:25.502391 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:25.502269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4d715528-b0d2-4eed-b118-396125ab82c7-tls-cert\") pod \"authorino-7ccdcbfdd6-8lv2s\" (UID: \"4d715528-b0d2-4eed-b118-396125ab82c7\") " pod="kuadrant-system/authorino-7ccdcbfdd6-8lv2s" Jul 12 18:29:25.504850 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:25.504816 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4d715528-b0d2-4eed-b118-396125ab82c7-tls-cert\") pod \"authorino-7ccdcbfdd6-8lv2s\" (UID: \"4d715528-b0d2-4eed-b118-396125ab82c7\") " pod="kuadrant-system/authorino-7ccdcbfdd6-8lv2s" Jul 12 18:29:25.510575 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:25.510551 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h49x2\" (UniqueName: \"kubernetes.io/projected/4d715528-b0d2-4eed-b118-396125ab82c7-kube-api-access-h49x2\") pod \"authorino-7ccdcbfdd6-8lv2s\" (UID: \"4d715528-b0d2-4eed-b118-396125ab82c7\") " pod="kuadrant-system/authorino-7ccdcbfdd6-8lv2s" Jul 12 18:29:25.541284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:25.541253 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ccdcbfdd6-8lv2s" Jul 12 18:29:25.587893 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:25.587860 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:29:25.680945 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:25.680912 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:29:25.686576 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:25.686544 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7ccdcbfdd6-8lv2s"] Jul 12 18:29:25.689855 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:29:25.689825 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d715528_b0d2_4eed_b118_396125ab82c7.slice/crio-a201fa73fdf1a861dc4bd66ab2a387f600f68be9ee5512173a70bba09fc5a69f WatchSource:0}: Error finding container a201fa73fdf1a861dc4bd66ab2a387f600f68be9ee5512173a70bba09fc5a69f: Status 404 returned error can't find the container with id a201fa73fdf1a861dc4bd66ab2a387f600f68be9ee5512173a70bba09fc5a69f Jul 12 18:29:25.691073 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:25.691053 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 18:29:26.239325 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:26.239283 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ccdcbfdd6-8lv2s" event={"ID":"4d715528-b0d2-4eed-b118-396125ab82c7","Type":"ContainerStarted","Data":"f35e030e8c8e6c6b3a51b7f8e422650664b354cded1ec19db830a64711db7c99"} Jul 12 18:29:26.239810 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:26.239339 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ccdcbfdd6-8lv2s" event={"ID":"4d715528-b0d2-4eed-b118-396125ab82c7","Type":"ContainerStarted","Data":"a201fa73fdf1a861dc4bd66ab2a387f600f68be9ee5512173a70bba09fc5a69f"} Jul 12 18:29:26.261392 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:26.261328 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7ccdcbfdd6-8lv2s" podStartSLOduration=0.868133332 podStartE2EDuration="1.261302889s" totalImagesPullingTime="393.169557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:29:26.259031859 +0000 UTC m=+728.026329441" watchObservedRunningTime="2026-07-12 18:29:26.261302889 +0000 UTC m=+728.028600473" Jul 12 18:29:26.696193 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:26.696143 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:29:26.794924 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:26.794891 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:29:28.818332 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:29:28.818297 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:30:00.140461 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:00.140423 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k"] Jul 12 18:30:00.144298 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:00.144278 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k" Jul 12 18:30:00.150464 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:00.150435 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k"] Jul 12 18:30:00.204591 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:00.204554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s2j9\" (UniqueName: \"kubernetes.io/projected/a4f0b600-0314-4baf-b422-7278ba0230b4-kube-api-access-8s2j9\") pod \"maas-api-key-cleanup-29731350-pj94k\" (UID: \"a4f0b600-0314-4baf-b422-7278ba0230b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k" Jul 12 18:30:00.305904 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:00.305861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s2j9\" (UniqueName: \"kubernetes.io/projected/a4f0b600-0314-4baf-b422-7278ba0230b4-kube-api-access-8s2j9\") pod \"maas-api-key-cleanup-29731350-pj94k\" (UID: \"a4f0b600-0314-4baf-b422-7278ba0230b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k" Jul 12 18:30:00.314299 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:00.314275 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s2j9\" (UniqueName: \"kubernetes.io/projected/a4f0b600-0314-4baf-b422-7278ba0230b4-kube-api-access-8s2j9\") pod \"maas-api-key-cleanup-29731350-pj94k\" (UID: \"a4f0b600-0314-4baf-b422-7278ba0230b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k" Jul 12 18:30:00.456437 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:00.456357 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k" Jul 12 18:30:00.604341 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:00.604308 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k"] Jul 12 18:30:00.606700 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:30:00.606660 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f0b600_0314_4baf_b422_7278ba0230b4.slice/crio-70a4c619ea1713fa4652e873eecfb4b91200bd9e7884e15fdd431c8a9fa790fb WatchSource:0}: Error finding container 70a4c619ea1713fa4652e873eecfb4b91200bd9e7884e15fdd431c8a9fa790fb: Status 404 returned error can't find the container with id 70a4c619ea1713fa4652e873eecfb4b91200bd9e7884e15fdd431c8a9fa790fb Jul 12 18:30:01.434291 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:01.434256 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k" event={"ID":"a4f0b600-0314-4baf-b422-7278ba0230b4","Type":"ContainerStarted","Data":"70a4c619ea1713fa4652e873eecfb4b91200bd9e7884e15fdd431c8a9fa790fb"} Jul 12 18:30:03.446827 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:03.446732 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k" event={"ID":"a4f0b600-0314-4baf-b422-7278ba0230b4","Type":"ContainerStarted","Data":"045774791ee34ec13d283a40ad036ded768742f60843b9d4ed40bcb1fee83aa9"} Jul 12 18:30:03.463480 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:03.463435 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k" podStartSLOduration=1.191086764 podStartE2EDuration="3.46341981s" totalImagesPullingTime="2.272333046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:30:03.461111049 +0000 UTC m=+765.228408631" watchObservedRunningTime="2026-07-12 18:30:03.46341981 +0000 UTC m=+765.230717392" Jul 12 18:30:36.314154 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:36.314115 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:30:36.457434 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:36.457402 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bcb555768-kn555"] Jul 12 18:30:36.457831 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:36.457774 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6bcb555768-kn555" podUID="17a161a7-e8f4-4521-9cbc-07e7608d7588" containerName="manager" containerID="cri-o://891f64cb0bc172d11003e3d7f8d4467f315e47608d2931ea384a2a3a7e5ee50e" gracePeriod=30 Jul 12 18:30:36.622115 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:36.622085 2576 generic.go:356] "Generic (PLEG): container finished" podID="17a161a7-e8f4-4521-9cbc-07e7608d7588" containerID="891f64cb0bc172d11003e3d7f8d4467f315e47608d2931ea384a2a3a7e5ee50e" exitCode=0 Jul 12 18:30:36.622273 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:36.622175 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bcb555768-kn555" event={"ID":"17a161a7-e8f4-4521-9cbc-07e7608d7588","Type":"ContainerDied","Data":"891f64cb0bc172d11003e3d7f8d4467f315e47608d2931ea384a2a3a7e5ee50e"} Jul 12 18:30:36.717205 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:36.717177 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bcb555768-kn555" Jul 12 18:30:36.850948 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:36.850865 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7clp\" (UniqueName: \"kubernetes.io/projected/17a161a7-e8f4-4521-9cbc-07e7608d7588-kube-api-access-z7clp\") pod \"17a161a7-e8f4-4521-9cbc-07e7608d7588\" (UID: \"17a161a7-e8f4-4521-9cbc-07e7608d7588\") " Jul 12 18:30:36.850948 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:36.850920 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17a161a7-e8f4-4521-9cbc-07e7608d7588-webhook-cert\") pod \"17a161a7-e8f4-4521-9cbc-07e7608d7588\" (UID: \"17a161a7-e8f4-4521-9cbc-07e7608d7588\") " Jul 12 18:30:36.853045 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:36.853010 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a161a7-e8f4-4521-9cbc-07e7608d7588-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "17a161a7-e8f4-4521-9cbc-07e7608d7588" (UID: "17a161a7-e8f4-4521-9cbc-07e7608d7588"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:30:36.853179 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:36.853083 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a161a7-e8f4-4521-9cbc-07e7608d7588-kube-api-access-z7clp" (OuterVolumeSpecName: "kube-api-access-z7clp") pod "17a161a7-e8f4-4521-9cbc-07e7608d7588" (UID: "17a161a7-e8f4-4521-9cbc-07e7608d7588"). InnerVolumeSpecName "kube-api-access-z7clp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:30:36.951754 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:36.951718 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7clp\" (UniqueName: \"kubernetes.io/projected/17a161a7-e8f4-4521-9cbc-07e7608d7588-kube-api-access-z7clp\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:30:36.951754 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:36.951749 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17a161a7-e8f4-4521-9cbc-07e7608d7588-webhook-cert\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:30:37.628112 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:37.628072 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bcb555768-kn555" event={"ID":"17a161a7-e8f4-4521-9cbc-07e7608d7588","Type":"ContainerDied","Data":"4c895b5da7e8c990fde73dea04f74cc5593c21eb78f21b6f34eb6f5521358152"} Jul 12 18:30:37.628112 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:37.628103 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bcb555768-kn555" Jul 12 18:30:37.628705 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:37.628138 2576 scope.go:117] "RemoveContainer" containerID="891f64cb0bc172d11003e3d7f8d4467f315e47608d2931ea384a2a3a7e5ee50e" Jul 12 18:30:37.651088 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:37.651058 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bcb555768-kn555"] Jul 12 18:30:37.653619 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:37.653593 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6bcb555768-kn555"] Jul 12 18:30:38.287934 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.287898 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:30:38.403260 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.403227 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6bcb555768-r7d65"] Jul 12 18:30:38.403731 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.403707 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a161a7-e8f4-4521-9cbc-07e7608d7588" containerName="manager" Jul 12 18:30:38.403731 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.403727 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a161a7-e8f4-4521-9cbc-07e7608d7588" containerName="manager" Jul 12 18:30:38.403982 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.403877 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a161a7-e8f4-4521-9cbc-07e7608d7588" containerName="manager" Jul 12 18:30:38.408775 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.408757 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bcb555768-r7d65" Jul 12 18:30:38.412708 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.412690 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jrjhq\"" Jul 12 18:30:38.412850 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.412692 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 12 18:30:38.412850 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.412692 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 12 18:30:38.417567 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.417543 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bcb555768-r7d65"] Jul 12 18:30:38.466560 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.466531 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbz9n\" (UniqueName: \"kubernetes.io/projected/4751ce85-56bc-4992-a7a8-ba1543377dc8-kube-api-access-lbz9n\") pod \"maas-controller-6bcb555768-r7d65\" (UID: \"4751ce85-56bc-4992-a7a8-ba1543377dc8\") " pod="opendatahub/maas-controller-6bcb555768-r7d65" Jul 12 18:30:38.466743 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.466595 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4751ce85-56bc-4992-a7a8-ba1543377dc8-webhook-cert\") pod \"maas-controller-6bcb555768-r7d65\" (UID: \"4751ce85-56bc-4992-a7a8-ba1543377dc8\") " pod="opendatahub/maas-controller-6bcb555768-r7d65" Jul 12 18:30:38.568180 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.568091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbz9n\" (UniqueName: \"kubernetes.io/projected/4751ce85-56bc-4992-a7a8-ba1543377dc8-kube-api-access-lbz9n\") pod \"maas-controller-6bcb555768-r7d65\" (UID: \"4751ce85-56bc-4992-a7a8-ba1543377dc8\") " pod="opendatahub/maas-controller-6bcb555768-r7d65" Jul 12 18:30:38.568180 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.568159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4751ce85-56bc-4992-a7a8-ba1543377dc8-webhook-cert\") pod \"maas-controller-6bcb555768-r7d65\" (UID: \"4751ce85-56bc-4992-a7a8-ba1543377dc8\") " pod="opendatahub/maas-controller-6bcb555768-r7d65" Jul 12 18:30:38.570733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.570700 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4751ce85-56bc-4992-a7a8-ba1543377dc8-webhook-cert\") pod \"maas-controller-6bcb555768-r7d65\" (UID: \"4751ce85-56bc-4992-a7a8-ba1543377dc8\") " pod="opendatahub/maas-controller-6bcb555768-r7d65" Jul 12 18:30:38.576589 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.576562 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbz9n\" (UniqueName: \"kubernetes.io/projected/4751ce85-56bc-4992-a7a8-ba1543377dc8-kube-api-access-lbz9n\") pod \"maas-controller-6bcb555768-r7d65\" (UID: \"4751ce85-56bc-4992-a7a8-ba1543377dc8\") " pod="opendatahub/maas-controller-6bcb555768-r7d65" Jul 12 18:30:38.719948 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.719916 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bcb555768-r7d65" Jul 12 18:30:38.830188 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.830139 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17a161a7-e8f4-4521-9cbc-07e7608d7588" path="/var/lib/kubelet/pods/17a161a7-e8f4-4521-9cbc-07e7608d7588/volumes" Jul 12 18:30:38.860023 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:38.859989 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bcb555768-r7d65"] Jul 12 18:30:38.862175 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:30:38.862150 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4751ce85_56bc_4992_a7a8_ba1543377dc8.slice/crio-7227285361c3c76b7294635d2a845653e9419c3d97d6c4b1ec5a19fa944bb372 WatchSource:0}: Error finding container 7227285361c3c76b7294635d2a845653e9419c3d97d6c4b1ec5a19fa944bb372: Status 404 returned error can't find the container with id 7227285361c3c76b7294635d2a845653e9419c3d97d6c4b1ec5a19fa944bb372 Jul 12 18:30:39.642331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:39.642298 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bcb555768-r7d65" event={"ID":"4751ce85-56bc-4992-a7a8-ba1543377dc8","Type":"ContainerStarted","Data":"8049efa42775b8dddcb16793315bbd7214c80cc38a6e258de12569754d706ac4"} Jul 12 18:30:39.642507 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:39.642352 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bcb555768-r7d65" event={"ID":"4751ce85-56bc-4992-a7a8-ba1543377dc8","Type":"ContainerStarted","Data":"7227285361c3c76b7294635d2a845653e9419c3d97d6c4b1ec5a19fa944bb372"} Jul 12 18:30:39.642507 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:39.642447 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6bcb555768-r7d65" Jul 12 18:30:39.658496 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:39.658321 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6bcb555768-r7d65" podStartSLOduration=1.210228521 podStartE2EDuration="1.658302235s" totalImagesPullingTime="448.073714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:30:39.658151256 +0000 UTC m=+801.425448840" watchObservedRunningTime="2026-07-12 18:30:39.658302235 +0000 UTC m=+801.425599817" Jul 12 18:30:47.238470 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:47.238431 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:30:50.652776 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:30:50.652746 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6bcb555768-r7d65" Jul 12 18:31:00.010836 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:00.010802 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k"] Jul 12 18:31:00.011234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:00.010981 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k" podUID="a4f0b600-0314-4baf-b422-7278ba0230b4" containerName="cleanup" containerID="cri-o://045774791ee34ec13d283a40ad036ded768742f60843b9d4ed40bcb1fee83aa9" gracePeriod=30 Jul 12 18:31:07.645737 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:07.645682 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:31:16.605361 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:16.605325 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:31:16.704012 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:16.703960 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:31:16.812744 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:16.812712 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:31:22.873428 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:22.873391 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:31:22.985038 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:22.985000 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:31:23.075265 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:23.075231 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:31:23.209064 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:23.208990 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:31:28.939769 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:28.939737 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:31:28.998290 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:28.998254 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:31:30.203912 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:30.203878 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k" Jul 12 18:31:30.360565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:30.360530 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s2j9\" (UniqueName: \"kubernetes.io/projected/a4f0b600-0314-4baf-b422-7278ba0230b4-kube-api-access-8s2j9\") pod \"a4f0b600-0314-4baf-b422-7278ba0230b4\" (UID: \"a4f0b600-0314-4baf-b422-7278ba0230b4\") " Jul 12 18:31:30.362835 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:30.362808 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f0b600-0314-4baf-b422-7278ba0230b4-kube-api-access-8s2j9" (OuterVolumeSpecName: "kube-api-access-8s2j9") pod "a4f0b600-0314-4baf-b422-7278ba0230b4" (UID: "a4f0b600-0314-4baf-b422-7278ba0230b4"). InnerVolumeSpecName "kube-api-access-8s2j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:31:30.462140 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:30.462090 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s2j9\" (UniqueName: \"kubernetes.io/projected/a4f0b600-0314-4baf-b422-7278ba0230b4-kube-api-access-8s2j9\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:31:30.912330 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:30.912300 2576 generic.go:356] "Generic (PLEG): container finished" podID="a4f0b600-0314-4baf-b422-7278ba0230b4" containerID="045774791ee34ec13d283a40ad036ded768742f60843b9d4ed40bcb1fee83aa9" exitCode=137 Jul 12 18:31:30.912520 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:30.912347 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k" Jul 12 18:31:30.912520 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:30.912380 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k" event={"ID":"a4f0b600-0314-4baf-b422-7278ba0230b4","Type":"ContainerDied","Data":"045774791ee34ec13d283a40ad036ded768742f60843b9d4ed40bcb1fee83aa9"} Jul 12 18:31:30.912520 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:30.912439 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k" event={"ID":"a4f0b600-0314-4baf-b422-7278ba0230b4","Type":"ContainerDied","Data":"70a4c619ea1713fa4652e873eecfb4b91200bd9e7884e15fdd431c8a9fa790fb"} Jul 12 18:31:30.912520 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:30.912466 2576 scope.go:117] "RemoveContainer" containerID="045774791ee34ec13d283a40ad036ded768742f60843b9d4ed40bcb1fee83aa9" Jul 12 18:31:30.921248 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:30.921230 2576 scope.go:117] "RemoveContainer" containerID="045774791ee34ec13d283a40ad036ded768742f60843b9d4ed40bcb1fee83aa9" Jul 12 18:31:30.921494 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:31:30.921472 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"045774791ee34ec13d283a40ad036ded768742f60843b9d4ed40bcb1fee83aa9\": container with ID starting with 045774791ee34ec13d283a40ad036ded768742f60843b9d4ed40bcb1fee83aa9 not found: ID does not exist" containerID="045774791ee34ec13d283a40ad036ded768742f60843b9d4ed40bcb1fee83aa9" Jul 12 18:31:30.921560 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:30.921506 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"045774791ee34ec13d283a40ad036ded768742f60843b9d4ed40bcb1fee83aa9"} err="failed to get container status \"045774791ee34ec13d283a40ad036ded768742f60843b9d4ed40bcb1fee83aa9\": rpc error: code = NotFound desc = could not find container \"045774791ee34ec13d283a40ad036ded768742f60843b9d4ed40bcb1fee83aa9\": container with ID starting with 045774791ee34ec13d283a40ad036ded768742f60843b9d4ed40bcb1fee83aa9 not found: ID does not exist" Jul 12 18:31:30.928664 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:30.928644 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k"] Jul 12 18:31:30.930538 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:30.930513 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731350-pj94k"] Jul 12 18:31:32.822663 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:31:32.822626 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4f0b600-0314-4baf-b422-7278ba0230b4" path="/var/lib/kubelet/pods/a4f0b600-0314-4baf-b422-7278ba0230b4/volumes" Jul 12 18:33:04.456298 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:33:04.456242 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:34:38.061714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:34:38.061682 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:34:38.159772 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:34:38.159734 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:35:24.530127 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:24.530038 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh"] Jul 12 18:35:24.530685 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:24.530237 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" podUID="a1e93a66-863f-45c1-9e29-01d6e22cb168" containerName="manager" containerID="cri-o://eccf4a1df5e98d212935efeff6f12d7b332a42c808faaa8ee44ac75441c10e2a" gracePeriod=10 Jul 12 18:35:27.295520 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:27.295494 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" Jul 12 18:35:27.457291 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:27.457211 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5q77\" (UniqueName: \"kubernetes.io/projected/a1e93a66-863f-45c1-9e29-01d6e22cb168-kube-api-access-c5q77\") pod \"a1e93a66-863f-45c1-9e29-01d6e22cb168\" (UID: \"a1e93a66-863f-45c1-9e29-01d6e22cb168\") " Jul 12 18:35:27.457291 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:27.457260 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1e93a66-863f-45c1-9e29-01d6e22cb168-extensions-socket-volume\") pod \"a1e93a66-863f-45c1-9e29-01d6e22cb168\" (UID: \"a1e93a66-863f-45c1-9e29-01d6e22cb168\") " Jul 12 18:35:27.457671 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:27.457637 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e93a66-863f-45c1-9e29-01d6e22cb168-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a1e93a66-863f-45c1-9e29-01d6e22cb168" (UID: "a1e93a66-863f-45c1-9e29-01d6e22cb168"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:35:27.459232 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:27.459211 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e93a66-863f-45c1-9e29-01d6e22cb168-kube-api-access-c5q77" (OuterVolumeSpecName: "kube-api-access-c5q77") pod "a1e93a66-863f-45c1-9e29-01d6e22cb168" (UID: "a1e93a66-863f-45c1-9e29-01d6e22cb168"). InnerVolumeSpecName "kube-api-access-c5q77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:35:27.557892 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:27.557858 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5q77\" (UniqueName: \"kubernetes.io/projected/a1e93a66-863f-45c1-9e29-01d6e22cb168-kube-api-access-c5q77\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:35:27.557892 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:27.557887 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1e93a66-863f-45c1-9e29-01d6e22cb168-extensions-socket-volume\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:35:28.148487 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:28.148453 2576 generic.go:356] "Generic (PLEG): container finished" podID="a1e93a66-863f-45c1-9e29-01d6e22cb168" containerID="eccf4a1df5e98d212935efeff6f12d7b332a42c808faaa8ee44ac75441c10e2a" exitCode=0 Jul 12 18:35:28.148674 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:28.148501 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" event={"ID":"a1e93a66-863f-45c1-9e29-01d6e22cb168","Type":"ContainerDied","Data":"eccf4a1df5e98d212935efeff6f12d7b332a42c808faaa8ee44ac75441c10e2a"} Jul 12 18:35:28.148674 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:28.148513 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" Jul 12 18:35:28.148674 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:28.148553 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh" event={"ID":"a1e93a66-863f-45c1-9e29-01d6e22cb168","Type":"ContainerDied","Data":"3d7508b43853264ac293036f631ae2e4597c7d356c8417240b6160b59fc2485b"} Jul 12 18:35:28.148674 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:28.148582 2576 scope.go:117] "RemoveContainer" containerID="eccf4a1df5e98d212935efeff6f12d7b332a42c808faaa8ee44ac75441c10e2a" Jul 12 18:35:28.158500 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:28.158479 2576 scope.go:117] "RemoveContainer" containerID="eccf4a1df5e98d212935efeff6f12d7b332a42c808faaa8ee44ac75441c10e2a" Jul 12 18:35:28.158744 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:35:28.158724 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eccf4a1df5e98d212935efeff6f12d7b332a42c808faaa8ee44ac75441c10e2a\": container with ID starting with eccf4a1df5e98d212935efeff6f12d7b332a42c808faaa8ee44ac75441c10e2a not found: ID does not exist" containerID="eccf4a1df5e98d212935efeff6f12d7b332a42c808faaa8ee44ac75441c10e2a" Jul 12 18:35:28.158826 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:28.158757 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eccf4a1df5e98d212935efeff6f12d7b332a42c808faaa8ee44ac75441c10e2a"} err="failed to get container status \"eccf4a1df5e98d212935efeff6f12d7b332a42c808faaa8ee44ac75441c10e2a\": rpc error: code = NotFound desc = could not find container \"eccf4a1df5e98d212935efeff6f12d7b332a42c808faaa8ee44ac75441c10e2a\": container with ID starting with eccf4a1df5e98d212935efeff6f12d7b332a42c808faaa8ee44ac75441c10e2a not found: ID does not exist" Jul 12 18:35:28.170518 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:28.170491 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh"] Jul 12 18:35:28.175436 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:28.175415 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sc6kh"] Jul 12 18:35:28.818479 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:35:28.818439 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1e93a66-863f-45c1-9e29-01d6e22cb168" path="/var/lib/kubelet/pods/a1e93a66-863f-45c1-9e29-01d6e22cb168/volumes" Jul 12 18:36:25.618767 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:25.618727 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gzpxp"] Jul 12 18:36:25.619191 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:25.619181 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f0b600-0314-4baf-b422-7278ba0230b4" containerName="cleanup" Jul 12 18:36:25.619233 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:25.619193 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f0b600-0314-4baf-b422-7278ba0230b4" containerName="cleanup" Jul 12 18:36:25.619233 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:25.619206 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1e93a66-863f-45c1-9e29-01d6e22cb168" containerName="manager" Jul 12 18:36:25.619233 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:25.619211 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1e93a66-863f-45c1-9e29-01d6e22cb168" containerName="manager" Jul 12 18:36:25.619323 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:25.619294 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1e93a66-863f-45c1-9e29-01d6e22cb168" containerName="manager" Jul 12 18:36:25.619323 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:25.619303 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4f0b600-0314-4baf-b422-7278ba0230b4" containerName="cleanup" Jul 12 18:36:25.622326 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:25.622311 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gzpxp" Jul 12 18:36:25.624871 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:25.624850 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gcg9j\"" Jul 12 18:36:25.636819 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:25.636798 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gzpxp"] Jul 12 18:36:25.694578 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:25.694545 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klrhl\" (UniqueName: \"kubernetes.io/projected/4679d403-b9fb-4cae-b5b3-9e297d5f8a5d-kube-api-access-klrhl\") pod \"kuadrant-operator-controller-manager-86878c769f-gzpxp\" (UID: \"4679d403-b9fb-4cae-b5b3-9e297d5f8a5d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gzpxp" Jul 12 18:36:25.694764 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:25.694608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4679d403-b9fb-4cae-b5b3-9e297d5f8a5d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gzpxp\" (UID: \"4679d403-b9fb-4cae-b5b3-9e297d5f8a5d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gzpxp" Jul 12 18:36:25.795948 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:25.795913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klrhl\" (UniqueName: \"kubernetes.io/projected/4679d403-b9fb-4cae-b5b3-9e297d5f8a5d-kube-api-access-klrhl\") pod \"kuadrant-operator-controller-manager-86878c769f-gzpxp\" (UID: \"4679d403-b9fb-4cae-b5b3-9e297d5f8a5d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gzpxp" Jul 12 18:36:25.796151 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:25.796003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4679d403-b9fb-4cae-b5b3-9e297d5f8a5d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gzpxp\" (UID: \"4679d403-b9fb-4cae-b5b3-9e297d5f8a5d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gzpxp" Jul 12 18:36:25.796379 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:25.796359 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4679d403-b9fb-4cae-b5b3-9e297d5f8a5d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gzpxp\" (UID: \"4679d403-b9fb-4cae-b5b3-9e297d5f8a5d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gzpxp" Jul 12 18:36:25.810730 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:25.810703 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klrhl\" (UniqueName: \"kubernetes.io/projected/4679d403-b9fb-4cae-b5b3-9e297d5f8a5d-kube-api-access-klrhl\") pod \"kuadrant-operator-controller-manager-86878c769f-gzpxp\" (UID: \"4679d403-b9fb-4cae-b5b3-9e297d5f8a5d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gzpxp" Jul 12 18:36:25.932749 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:25.932656 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gzpxp" Jul 12 18:36:26.073127 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:26.073091 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gzpxp"] Jul 12 18:36:26.076833 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:36:26.076804 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4679d403_b9fb_4cae_b5b3_9e297d5f8a5d.slice/crio-94436cff7cd3d9cd3994017fad33f9301c3af955fb945953a88ea075f01dfc9b WatchSource:0}: Error finding container 94436cff7cd3d9cd3994017fad33f9301c3af955fb945953a88ea075f01dfc9b: Status 404 returned error can't find the container with id 94436cff7cd3d9cd3994017fad33f9301c3af955fb945953a88ea075f01dfc9b Jul 12 18:36:26.437484 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:26.437450 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gzpxp" event={"ID":"4679d403-b9fb-4cae-b5b3-9e297d5f8a5d","Type":"ContainerStarted","Data":"e5d4e263a780988817279eb8413619172f117bfeb778ae52a0e7530cb29aff28"} Jul 12 18:36:26.437654 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:26.437511 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gzpxp" Jul 12 18:36:26.437654 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:26.437528 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gzpxp" event={"ID":"4679d403-b9fb-4cae-b5b3-9e297d5f8a5d","Type":"ContainerStarted","Data":"94436cff7cd3d9cd3994017fad33f9301c3af955fb945953a88ea075f01dfc9b"} Jul 12 18:36:26.459993 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:26.459934 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gzpxp" podStartSLOduration=1.4599127570000001 podStartE2EDuration="1.459912757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:36:26.458185127 +0000 UTC m=+1148.225482711" watchObservedRunningTime="2026-07-12 18:36:26.459912757 +0000 UTC m=+1148.227210339" Jul 12 18:36:37.444161 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:36:37.444127 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gzpxp" Jul 12 18:37:38.207203 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:37:38.207168 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:38:45.664511 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:38:45.664397 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:38:46.823002 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:38:46.822972 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:39:40.377163 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:39:40.377081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:39:40.432681 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:39:40.432631 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:08.843977 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:08.843940 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:20.346360 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:20.346322 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:27.767961 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:27.767914 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:36.031072 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:36.031032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:41.102452 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:41.102358 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:43.849648 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:43.849595 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:43.949632 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:43.949597 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:44.285487 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:44.285401 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:49.942153 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:49.942120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:50.043601 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.043568 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:50.131047 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.131010 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj"] Jul 12 18:42:50.135485 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.135459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.137945 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.137923 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bcf92e9a-openshift-default-dockercfg-jjv7f\"" Jul 12 18:42:50.150619 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.150589 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj"] Jul 12 18:42:50.159039 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.159006 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:50.216511 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.216419 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-workload-socket\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.216511 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.216466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-podinfo\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.216696 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.216523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-credential-socket\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.216696 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.216540 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-token\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.216696 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.216583 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-workload-certs\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.216696 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.216610 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfnzz\" (UniqueName: \"kubernetes.io/projected/d3137d19-e8d9-4f71-9ba8-570b146171cc-kube-api-access-kfnzz\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.216696 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.216635 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-data\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.216909 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.216705 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-envoy\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.216909 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.216731 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3137d19-e8d9-4f71-9ba8-570b146171cc-istiod-ca-cert\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.259084 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.259052 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:50.318062 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.318033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-envoy\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.318225 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.318072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3137d19-e8d9-4f71-9ba8-570b146171cc-istiod-ca-cert\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.318225 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.318114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-workload-socket\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.318225 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.318132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-podinfo\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.318225 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.318161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-credential-socket\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.318225 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.318181 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-token\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.318225 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.318216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-workload-certs\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.318518 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.318257 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfnzz\" (UniqueName: \"kubernetes.io/projected/d3137d19-e8d9-4f71-9ba8-570b146171cc-kube-api-access-kfnzz\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.318518 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.318292 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-data\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.318630 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.318607 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-workload-socket\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.318692 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.318613 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-data\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.318931 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.318901 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-workload-certs\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.319048 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.318981 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-credential-socket\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.319334 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.319307 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3137d19-e8d9-4f71-9ba8-570b146171cc-istiod-ca-cert\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.320543 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.320522 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-envoy\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.320937 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.320917 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-podinfo\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.326856 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.326828 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-token\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.326951 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.326915 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfnzz\" (UniqueName: \"kubernetes.io/projected/d3137d19-e8d9-4f71-9ba8-570b146171cc-kube-api-access-kfnzz\") pod \"e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.446987 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.446947 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:50.589528 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.589503 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj"] Jul 12 18:42:50.591310 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:42:50.591285 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3137d19_e8d9_4f71_9ba8_570b146171cc.slice/crio-69b89ceed53e100e2b8e645c8a50f72e6218cc7a0c0296ff8be102c5c9ac46a5 WatchSource:0}: Error finding container 69b89ceed53e100e2b8e645c8a50f72e6218cc7a0c0296ff8be102c5c9ac46a5: Status 404 returned error can't find the container with id 69b89ceed53e100e2b8e645c8a50f72e6218cc7a0c0296ff8be102c5c9ac46a5 Jul 12 18:42:50.593473 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.593442 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:42:50.593550 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.593509 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:42:50.593550 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:50.593538 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:42:51.247966 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.247929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:51.324961 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.324919 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc"] Jul 12 18:42:51.328170 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.328141 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:51.329154 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.329133 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" Jul 12 18:42:51.332135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.332112 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bcf92e9a-serving-cert\"" Jul 12 18:42:51.339037 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.338697 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc"] Jul 12 18:42:51.396614 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.396581 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:51.425983 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.425948 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" event={"ID":"d3137d19-e8d9-4f71-9ba8-570b146171cc","Type":"ContainerStarted","Data":"d2bf20ac31802c3b6e850e775507d1d7882eb762dfdea4d1a3b1d8933a8a80f2"} Jul 12 18:42:51.426212 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.426194 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" event={"ID":"d3137d19-e8d9-4f71-9ba8-570b146171cc","Type":"ContainerStarted","Data":"69b89ceed53e100e2b8e645c8a50f72e6218cc7a0c0296ff8be102c5c9ac46a5"} Jul 12 18:42:51.427962 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.427923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35ff610d-8ea8-4003-b06b-af23256de55b-maas-api-tls\") pod \"maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc\" (UID: \"35ff610d-8ea8-4003-b06b-af23256de55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" Jul 12 18:42:51.428089 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.427987 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35ff610d-8ea8-4003-b06b-af23256de55b-openshift-service-ca\") pod \"maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc\" (UID: \"35ff610d-8ea8-4003-b06b-af23256de55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" Jul 12 18:42:51.428178 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.428148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tk2z\" (UniqueName: \"kubernetes.io/projected/35ff610d-8ea8-4003-b06b-af23256de55b-kube-api-access-8tk2z\") pod \"maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc\" (UID: \"35ff610d-8ea8-4003-b06b-af23256de55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" Jul 12 18:42:51.447877 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.447832 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:51.451311 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.451258 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" podStartSLOduration=1.451239554 podStartE2EDuration="1.451239554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:42:51.451040183 +0000 UTC m=+1533.218337766" watchObservedRunningTime="2026-07-12 18:42:51.451239554 +0000 UTC m=+1533.218537138" Jul 12 18:42:51.453213 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.453194 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:51.528794 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.528695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tk2z\" (UniqueName: \"kubernetes.io/projected/35ff610d-8ea8-4003-b06b-af23256de55b-kube-api-access-8tk2z\") pod \"maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc\" (UID: \"35ff610d-8ea8-4003-b06b-af23256de55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" Jul 12 18:42:51.528794 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.528760 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35ff610d-8ea8-4003-b06b-af23256de55b-maas-api-tls\") pod \"maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc\" (UID: \"35ff610d-8ea8-4003-b06b-af23256de55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" Jul 12 18:42:51.529032 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.528918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35ff610d-8ea8-4003-b06b-af23256de55b-openshift-service-ca\") pod \"maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc\" (UID: \"35ff610d-8ea8-4003-b06b-af23256de55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" Jul 12 18:42:51.529673 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.529650 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35ff610d-8ea8-4003-b06b-af23256de55b-openshift-service-ca\") pod \"maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc\" (UID: \"35ff610d-8ea8-4003-b06b-af23256de55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" Jul 12 18:42:51.529845 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.529774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:51.531307 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.531287 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35ff610d-8ea8-4003-b06b-af23256de55b-maas-api-tls\") pod \"maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc\" (UID: \"35ff610d-8ea8-4003-b06b-af23256de55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" Jul 12 18:42:51.537211 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.537192 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tk2z\" (UniqueName: \"kubernetes.io/projected/35ff610d-8ea8-4003-b06b-af23256de55b-kube-api-access-8tk2z\") pod \"maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc\" (UID: \"35ff610d-8ea8-4003-b06b-af23256de55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" Jul 12 18:42:51.642928 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.642894 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" Jul 12 18:42:51.786621 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.786593 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc"] Jul 12 18:42:51.788091 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:42:51.788034 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ff610d_8ea8_4003_b06b_af23256de55b.slice/crio-da809e266b4de8feee8e686fcd297e26c07a6a6f2dcc441af665bfc58514fdfe WatchSource:0}: Error finding container da809e266b4de8feee8e686fcd297e26c07a6a6f2dcc441af665bfc58514fdfe: Status 404 returned error can't find the container with id da809e266b4de8feee8e686fcd297e26c07a6a6f2dcc441af665bfc58514fdfe Jul 12 18:42:51.789435 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.789419 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 18:42:51.887050 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:51.887022 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:52.019045 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:52.019013 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:52.434492 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:52.434442 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" event={"ID":"35ff610d-8ea8-4003-b06b-af23256de55b","Type":"ContainerStarted","Data":"13e331db021843c54eef9041545731cc9d5653faf2b952c0e3295fdac84c5891"} Jul 12 18:42:52.435039 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:52.434692 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:52.435039 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:52.434736 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" event={"ID":"35ff610d-8ea8-4003-b06b-af23256de55b","Type":"ContainerStarted","Data":"da809e266b4de8feee8e686fcd297e26c07a6a6f2dcc441af665bfc58514fdfe"} Jul 12 18:42:52.435662 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:52.435605 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:42:52.450369 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:52.450323 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" podStartSLOduration=1.045742482 podStartE2EDuration="1.450305998s" totalImagesPullingTime="404.563516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:42:52.449930448 +0000 UTC m=+1534.217228031" watchObservedRunningTime="2026-07-12 18:42:52.450305998 +0000 UTC m=+1534.217603580" Jul 12 18:42:53.441178 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:53.441146 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" Jul 12 18:42:53.889166 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:53.889127 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:56.347827 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:56.346290 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:56.511127 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:56.511094 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj"] Jul 12 18:42:56.511378 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:56.511350 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" podUID="d3137d19-e8d9-4f71-9ba8-570b146171cc" containerName="istio-proxy" containerID="cri-o://d2bf20ac31802c3b6e850e775507d1d7882eb762dfdea4d1a3b1d8933a8a80f2" gracePeriod=30 Jul 12 18:42:56.524034 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:56.524004 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:56.630030 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:56.629925 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:56.704768 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:56.704735 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc"] Jul 12 18:42:56.704995 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:56.704963 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" podUID="35ff610d-8ea8-4003-b06b-af23256de55b" containerName="maas-api" containerID="cri-o://13e331db021843c54eef9041545731cc9d5653faf2b952c0e3295fdac84c5891" gracePeriod=30 Jul 12 18:42:56.732360 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:56.732255 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:57.014683 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.014657 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" Jul 12 18:42:57.082094 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.082057 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35ff610d-8ea8-4003-b06b-af23256de55b-maas-api-tls\") pod \"35ff610d-8ea8-4003-b06b-af23256de55b\" (UID: \"35ff610d-8ea8-4003-b06b-af23256de55b\") " Jul 12 18:42:57.082247 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.082130 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tk2z\" (UniqueName: \"kubernetes.io/projected/35ff610d-8ea8-4003-b06b-af23256de55b-kube-api-access-8tk2z\") pod \"35ff610d-8ea8-4003-b06b-af23256de55b\" (UID: \"35ff610d-8ea8-4003-b06b-af23256de55b\") " Jul 12 18:42:57.082303 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.082244 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35ff610d-8ea8-4003-b06b-af23256de55b-openshift-service-ca\") pod \"35ff610d-8ea8-4003-b06b-af23256de55b\" (UID: \"35ff610d-8ea8-4003-b06b-af23256de55b\") " Jul 12 18:42:57.082557 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.082535 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35ff610d-8ea8-4003-b06b-af23256de55b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "35ff610d-8ea8-4003-b06b-af23256de55b" (UID: "35ff610d-8ea8-4003-b06b-af23256de55b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:42:57.084291 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.084259 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ff610d-8ea8-4003-b06b-af23256de55b-kube-api-access-8tk2z" (OuterVolumeSpecName: "kube-api-access-8tk2z") pod "35ff610d-8ea8-4003-b06b-af23256de55b" (UID: "35ff610d-8ea8-4003-b06b-af23256de55b"). InnerVolumeSpecName "kube-api-access-8tk2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:42:57.084412 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.084309 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35ff610d-8ea8-4003-b06b-af23256de55b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "35ff610d-8ea8-4003-b06b-af23256de55b" (UID: "35ff610d-8ea8-4003-b06b-af23256de55b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:42:57.183775 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.183688 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35ff610d-8ea8-4003-b06b-af23256de55b-openshift-service-ca\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:42:57.183775 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.183721 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35ff610d-8ea8-4003-b06b-af23256de55b-maas-api-tls\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:42:57.183775 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.183730 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tk2z\" (UniqueName: \"kubernetes.io/projected/35ff610d-8ea8-4003-b06b-af23256de55b-kube-api-access-8tk2z\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:42:57.463933 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.463846 2576 generic.go:356] "Generic (PLEG): container finished" podID="35ff610d-8ea8-4003-b06b-af23256de55b" containerID="13e331db021843c54eef9041545731cc9d5653faf2b952c0e3295fdac84c5891" exitCode=0 Jul 12 18:42:57.463933 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.463890 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" Jul 12 18:42:57.464389 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.463893 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" event={"ID":"35ff610d-8ea8-4003-b06b-af23256de55b","Type":"ContainerDied","Data":"13e331db021843c54eef9041545731cc9d5653faf2b952c0e3295fdac84c5891"} Jul 12 18:42:57.464389 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.464039 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc" event={"ID":"35ff610d-8ea8-4003-b06b-af23256de55b","Type":"ContainerDied","Data":"da809e266b4de8feee8e686fcd297e26c07a6a6f2dcc441af665bfc58514fdfe"} Jul 12 18:42:57.464389 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.464075 2576 scope.go:117] "RemoveContainer" containerID="13e331db021843c54eef9041545731cc9d5653faf2b952c0e3295fdac84c5891" Jul 12 18:42:57.473490 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.473472 2576 scope.go:117] "RemoveContainer" containerID="13e331db021843c54eef9041545731cc9d5653faf2b952c0e3295fdac84c5891" Jul 12 18:42:57.473730 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:42:57.473711 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e331db021843c54eef9041545731cc9d5653faf2b952c0e3295fdac84c5891\": container with ID starting with 13e331db021843c54eef9041545731cc9d5653faf2b952c0e3295fdac84c5891 not found: ID does not exist" containerID="13e331db021843c54eef9041545731cc9d5653faf2b952c0e3295fdac84c5891" Jul 12 18:42:57.473804 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.473738 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e331db021843c54eef9041545731cc9d5653faf2b952c0e3295fdac84c5891"} err="failed to get container status \"13e331db021843c54eef9041545731cc9d5653faf2b952c0e3295fdac84c5891\": rpc error: code = NotFound desc = could not find container \"13e331db021843c54eef9041545731cc9d5653faf2b952c0e3295fdac84c5891\": container with ID starting with 13e331db021843c54eef9041545731cc9d5653faf2b952c0e3295fdac84c5891 not found: ID does not exist" Jul 12 18:42:57.484963 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.484941 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc"] Jul 12 18:42:57.491091 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:57.491073 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bcf92e9a-78489dd8df-fkmvc"] Jul 12 18:42:58.396860 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:58.396831 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:42:58.818617 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:42:58.818534 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35ff610d-8ea8-4003-b06b-af23256de55b" path="/var/lib/kubelet/pods/35ff610d-8ea8-4003-b06b-af23256de55b/volumes" Jul 12 18:43:02.260481 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:02.260450 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:02.361435 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:02.361400 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:02.469761 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:02.469721 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:02.535474 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:02.535384 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:03.390708 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.390681 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:43:03.442745 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.442667 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-credential-socket\") pod \"d3137d19-e8d9-4f71-9ba8-570b146171cc\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " Jul 12 18:43:03.442745 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.442708 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-podinfo\") pod \"d3137d19-e8d9-4f71-9ba8-570b146171cc\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " Jul 12 18:43:03.442954 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.442753 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-data\") pod \"d3137d19-e8d9-4f71-9ba8-570b146171cc\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " Jul 12 18:43:03.442954 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.442773 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfnzz\" (UniqueName: \"kubernetes.io/projected/d3137d19-e8d9-4f71-9ba8-570b146171cc-kube-api-access-kfnzz\") pod \"d3137d19-e8d9-4f71-9ba8-570b146171cc\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " Jul 12 18:43:03.442954 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.442863 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-envoy\") pod \"d3137d19-e8d9-4f71-9ba8-570b146171cc\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " Jul 12 18:43:03.442954 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.442897 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-workload-socket\") pod \"d3137d19-e8d9-4f71-9ba8-570b146171cc\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " Jul 12 18:43:03.443135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.442957 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d3137d19-e8d9-4f71-9ba8-570b146171cc" (UID: "d3137d19-e8d9-4f71-9ba8-570b146171cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:43:03.443135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.443013 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-workload-certs\") pod \"d3137d19-e8d9-4f71-9ba8-570b146171cc\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " Jul 12 18:43:03.443135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.443098 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-token\") pod \"d3137d19-e8d9-4f71-9ba8-570b146171cc\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " Jul 12 18:43:03.443135 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.443134 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3137d19-e8d9-4f71-9ba8-570b146171cc-istiod-ca-cert\") pod \"d3137d19-e8d9-4f71-9ba8-570b146171cc\" (UID: \"d3137d19-e8d9-4f71-9ba8-570b146171cc\") " Jul 12 18:43:03.443383 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.443156 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d3137d19-e8d9-4f71-9ba8-570b146171cc" (UID: "d3137d19-e8d9-4f71-9ba8-570b146171cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:43:03.443383 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.443305 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d3137d19-e8d9-4f71-9ba8-570b146171cc" (UID: "d3137d19-e8d9-4f71-9ba8-570b146171cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:43:03.443555 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.443530 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3137d19-e8d9-4f71-9ba8-570b146171cc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d3137d19-e8d9-4f71-9ba8-570b146171cc" (UID: "d3137d19-e8d9-4f71-9ba8-570b146171cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:43:03.443747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.443693 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-data" (OuterVolumeSpecName: "istio-data") pod "d3137d19-e8d9-4f71-9ba8-570b146171cc" (UID: "d3137d19-e8d9-4f71-9ba8-570b146171cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:43:03.443747 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.443564 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-credential-socket\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:43:03.443900 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.443764 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-workload-socket\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:43:03.443900 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.443807 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-workload-certs\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:43:03.445172 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.445145 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d3137d19-e8d9-4f71-9ba8-570b146171cc" (UID: "d3137d19-e8d9-4f71-9ba8-570b146171cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 18:43:03.445262 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.445191 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3137d19-e8d9-4f71-9ba8-570b146171cc-kube-api-access-kfnzz" (OuterVolumeSpecName: "kube-api-access-kfnzz") pod "d3137d19-e8d9-4f71-9ba8-570b146171cc" (UID: "d3137d19-e8d9-4f71-9ba8-570b146171cc"). InnerVolumeSpecName "kube-api-access-kfnzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:43:03.445405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.445379 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d3137d19-e8d9-4f71-9ba8-570b146171cc" (UID: "d3137d19-e8d9-4f71-9ba8-570b146171cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:43:03.445405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.445391 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-token" (OuterVolumeSpecName: "istio-token") pod "d3137d19-e8d9-4f71-9ba8-570b146171cc" (UID: "d3137d19-e8d9-4f71-9ba8-570b146171cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:43:03.503953 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.503910 2576 generic.go:356] "Generic (PLEG): container finished" podID="d3137d19-e8d9-4f71-9ba8-570b146171cc" containerID="d2bf20ac31802c3b6e850e775507d1d7882eb762dfdea4d1a3b1d8933a8a80f2" exitCode=0 Jul 12 18:43:03.504131 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.503977 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" Jul 12 18:43:03.504131 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.503997 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" event={"ID":"d3137d19-e8d9-4f71-9ba8-570b146171cc","Type":"ContainerDied","Data":"d2bf20ac31802c3b6e850e775507d1d7882eb762dfdea4d1a3b1d8933a8a80f2"} Jul 12 18:43:03.504131 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.504089 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj" event={"ID":"d3137d19-e8d9-4f71-9ba8-570b146171cc","Type":"ContainerDied","Data":"69b89ceed53e100e2b8e645c8a50f72e6218cc7a0c0296ff8be102c5c9ac46a5"} Jul 12 18:43:03.504131 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.504119 2576 scope.go:117] "RemoveContainer" containerID="d2bf20ac31802c3b6e850e775507d1d7882eb762dfdea4d1a3b1d8933a8a80f2" Jul 12 18:43:03.515855 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.515832 2576 scope.go:117] "RemoveContainer" containerID="d2bf20ac31802c3b6e850e775507d1d7882eb762dfdea4d1a3b1d8933a8a80f2" Jul 12 18:43:03.516163 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:43:03.516139 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2bf20ac31802c3b6e850e775507d1d7882eb762dfdea4d1a3b1d8933a8a80f2\": container with ID starting with d2bf20ac31802c3b6e850e775507d1d7882eb762dfdea4d1a3b1d8933a8a80f2 not found: ID does not exist" containerID="d2bf20ac31802c3b6e850e775507d1d7882eb762dfdea4d1a3b1d8933a8a80f2" Jul 12 18:43:03.516212 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.516175 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2bf20ac31802c3b6e850e775507d1d7882eb762dfdea4d1a3b1d8933a8a80f2"} err="failed to get container status \"d2bf20ac31802c3b6e850e775507d1d7882eb762dfdea4d1a3b1d8933a8a80f2\": rpc error: code = NotFound desc = could not find container \"d2bf20ac31802c3b6e850e775507d1d7882eb762dfdea4d1a3b1d8933a8a80f2\": container with ID starting with d2bf20ac31802c3b6e850e775507d1d7882eb762dfdea4d1a3b1d8933a8a80f2 not found: ID does not exist" Jul 12 18:43:03.535913 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.535846 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj"] Jul 12 18:43:03.538899 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.538873 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bcf92e9a-openshift-default-65db8b7ff7-k9khj"] Jul 12 18:43:03.544317 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.544296 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-envoy\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:43:03.544317 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.544318 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-token\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:43:03.544450 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.544335 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3137d19-e8d9-4f71-9ba8-570b146171cc-istiod-ca-cert\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:43:03.544450 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.544344 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-podinfo\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:43:03.544450 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.544352 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3137d19-e8d9-4f71-9ba8-570b146171cc-istio-data\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:43:03.544450 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.544361 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfnzz\" (UniqueName: \"kubernetes.io/projected/d3137d19-e8d9-4f71-9ba8-570b146171cc-kube-api-access-kfnzz\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:43:03.557172 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.557133 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:03.657625 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.657597 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:03.852221 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.852184 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b"] Jul 12 18:43:03.852650 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.852635 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35ff610d-8ea8-4003-b06b-af23256de55b" containerName="maas-api" Jul 12 18:43:03.852765 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.852652 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="35ff610d-8ea8-4003-b06b-af23256de55b" containerName="maas-api" Jul 12 18:43:03.852765 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.852667 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3137d19-e8d9-4f71-9ba8-570b146171cc" containerName="istio-proxy" Jul 12 18:43:03.852765 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.852673 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3137d19-e8d9-4f71-9ba8-570b146171cc" containerName="istio-proxy" Jul 12 18:43:03.852765 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.852735 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="35ff610d-8ea8-4003-b06b-af23256de55b" containerName="maas-api" Jul 12 18:43:03.852765 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.852745 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3137d19-e8d9-4f71-9ba8-570b146171cc" containerName="istio-proxy" Jul 12 18:43:03.857208 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.857188 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" Jul 12 18:43:03.860584 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.860569 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b7e51ce2-serving-cert\"" Jul 12 18:43:03.877174 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.877148 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b"] Jul 12 18:43:03.906600 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.906569 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:03.949512 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.949478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2953010-656d-4a10-a530-a36ae0b215dc-openshift-service-ca\") pod \"maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b\" (UID: \"b2953010-656d-4a10-a530-a36ae0b215dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" Jul 12 18:43:03.949653 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.949537 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25lgk\" (UniqueName: \"kubernetes.io/projected/b2953010-656d-4a10-a530-a36ae0b215dc-kube-api-access-25lgk\") pod \"maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b\" (UID: \"b2953010-656d-4a10-a530-a36ae0b215dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" Jul 12 18:43:03.949711 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:03.949651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2953010-656d-4a10-a530-a36ae0b215dc-maas-api-tls\") pod \"maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b\" (UID: \"b2953010-656d-4a10-a530-a36ae0b215dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" Jul 12 18:43:04.050930 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:04.050890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2953010-656d-4a10-a530-a36ae0b215dc-maas-api-tls\") pod \"maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b\" (UID: \"b2953010-656d-4a10-a530-a36ae0b215dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" Jul 12 18:43:04.051097 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:04.050968 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2953010-656d-4a10-a530-a36ae0b215dc-openshift-service-ca\") pod \"maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b\" (UID: \"b2953010-656d-4a10-a530-a36ae0b215dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" Jul 12 18:43:04.051097 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:04.051013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25lgk\" (UniqueName: \"kubernetes.io/projected/b2953010-656d-4a10-a530-a36ae0b215dc-kube-api-access-25lgk\") pod \"maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b\" (UID: \"b2953010-656d-4a10-a530-a36ae0b215dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" Jul 12 18:43:04.051097 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:43:04.051048 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-serving-cert: secret "maas-api-e2e-ait-b7e51ce2-serving-cert" not found Jul 12 18:43:04.051227 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:43:04.051125 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2953010-656d-4a10-a530-a36ae0b215dc-maas-api-tls podName:b2953010-656d-4a10-a530-a36ae0b215dc nodeName:}" failed. No retries permitted until 2026-07-12 18:43:04.551103302 +0000 UTC m=+1546.318400878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b2953010-656d-4a10-a530-a36ae0b215dc-maas-api-tls") pod "maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" (UID: "b2953010-656d-4a10-a530-a36ae0b215dc") : secret "maas-api-e2e-ait-b7e51ce2-serving-cert" not found Jul 12 18:43:04.051707 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:04.051680 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2953010-656d-4a10-a530-a36ae0b215dc-openshift-service-ca\") pod \"maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b\" (UID: \"b2953010-656d-4a10-a530-a36ae0b215dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" Jul 12 18:43:04.059794 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:04.059770 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25lgk\" (UniqueName: \"kubernetes.io/projected/b2953010-656d-4a10-a530-a36ae0b215dc-kube-api-access-25lgk\") pod \"maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b\" (UID: \"b2953010-656d-4a10-a530-a36ae0b215dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" Jul 12 18:43:04.288055 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:04.287947 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:04.391355 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:04.391320 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:04.491854 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:04.491820 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:04.556198 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:04.556102 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2953010-656d-4a10-a530-a36ae0b215dc-maas-api-tls\") pod \"maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b\" (UID: \"b2953010-656d-4a10-a530-a36ae0b215dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" Jul 12 18:43:04.558726 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:04.558703 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2953010-656d-4a10-a530-a36ae0b215dc-maas-api-tls\") pod \"maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b\" (UID: \"b2953010-656d-4a10-a530-a36ae0b215dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" Jul 12 18:43:04.767317 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:04.767268 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" Jul 12 18:43:04.821117 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:04.821019 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3137d19-e8d9-4f71-9ba8-570b146171cc" path="/var/lib/kubelet/pods/d3137d19-e8d9-4f71-9ba8-570b146171cc/volumes" Jul 12 18:43:04.919973 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:04.919949 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b"] Jul 12 18:43:04.921695 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:43:04.921654 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2953010_656d_4a10_a530_a36ae0b215dc.slice/crio-b9e799fde329db0139ce18494166722810bea9408bd1e8e95f14cb6cf0580488 WatchSource:0}: Error finding container b9e799fde329db0139ce18494166722810bea9408bd1e8e95f14cb6cf0580488: Status 404 returned error can't find the container with id b9e799fde329db0139ce18494166722810bea9408bd1e8e95f14cb6cf0580488 Jul 12 18:43:05.516840 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:05.516715 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" event={"ID":"b2953010-656d-4a10-a530-a36ae0b215dc","Type":"ContainerStarted","Data":"14436c37d3bc4d35e7831a8f5efd028b6a132ddedfb03b78665caa3358cc0b49"} Jul 12 18:43:05.516840 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:05.516769 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" event={"ID":"b2953010-656d-4a10-a530-a36ae0b215dc","Type":"ContainerStarted","Data":"b9e799fde329db0139ce18494166722810bea9408bd1e8e95f14cb6cf0580488"} Jul 12 18:43:05.516840 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:05.516818 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" Jul 12 18:43:05.532845 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:05.532771 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" podStartSLOduration=2.204284032 podStartE2EDuration="2.532750646s" totalImagesPullingTime="328.466614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:43:05.531896459 +0000 UTC m=+1547.299194042" watchObservedRunningTime="2026-07-12 18:43:05.532750646 +0000 UTC m=+1547.300048229" Jul 12 18:43:05.572293 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:05.572247 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:08.655444 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:08.655408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:08.952770 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:08.952097 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b"] Jul 12 18:43:08.952770 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:08.952340 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" podUID="b2953010-656d-4a10-a530-a36ae0b215dc" containerName="maas-api" containerID="cri-o://14436c37d3bc4d35e7831a8f5efd028b6a132ddedfb03b78665caa3358cc0b49" gracePeriod=30 Jul 12 18:43:08.974689 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:08.974654 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:09.203333 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.203268 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" Jul 12 18:43:09.303588 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.303548 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2953010-656d-4a10-a530-a36ae0b215dc-maas-api-tls\") pod \"b2953010-656d-4a10-a530-a36ae0b215dc\" (UID: \"b2953010-656d-4a10-a530-a36ae0b215dc\") " Jul 12 18:43:09.303769 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.303652 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25lgk\" (UniqueName: \"kubernetes.io/projected/b2953010-656d-4a10-a530-a36ae0b215dc-kube-api-access-25lgk\") pod \"b2953010-656d-4a10-a530-a36ae0b215dc\" (UID: \"b2953010-656d-4a10-a530-a36ae0b215dc\") " Jul 12 18:43:09.303769 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.303696 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2953010-656d-4a10-a530-a36ae0b215dc-openshift-service-ca\") pod \"b2953010-656d-4a10-a530-a36ae0b215dc\" (UID: \"b2953010-656d-4a10-a530-a36ae0b215dc\") " Jul 12 18:43:09.304129 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.304102 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2953010-656d-4a10-a530-a36ae0b215dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b2953010-656d-4a10-a530-a36ae0b215dc" (UID: "b2953010-656d-4a10-a530-a36ae0b215dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:43:09.305674 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.305647 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2953010-656d-4a10-a530-a36ae0b215dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b2953010-656d-4a10-a530-a36ae0b215dc" (UID: "b2953010-656d-4a10-a530-a36ae0b215dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:43:09.305878 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.305849 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2953010-656d-4a10-a530-a36ae0b215dc-kube-api-access-25lgk" (OuterVolumeSpecName: "kube-api-access-25lgk") pod "b2953010-656d-4a10-a530-a36ae0b215dc" (UID: "b2953010-656d-4a10-a530-a36ae0b215dc"). InnerVolumeSpecName "kube-api-access-25lgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:43:09.405224 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.405188 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25lgk\" (UniqueName: \"kubernetes.io/projected/b2953010-656d-4a10-a530-a36ae0b215dc-kube-api-access-25lgk\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:43:09.405224 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.405220 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2953010-656d-4a10-a530-a36ae0b215dc-openshift-service-ca\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:43:09.405437 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.405235 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2953010-656d-4a10-a530-a36ae0b215dc-maas-api-tls\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:43:09.538768 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.538672 2576 generic.go:356] "Generic (PLEG): container finished" podID="b2953010-656d-4a10-a530-a36ae0b215dc" containerID="14436c37d3bc4d35e7831a8f5efd028b6a132ddedfb03b78665caa3358cc0b49" exitCode=0 Jul 12 18:43:09.538768 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.538724 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" Jul 12 18:43:09.538768 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.538753 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" event={"ID":"b2953010-656d-4a10-a530-a36ae0b215dc","Type":"ContainerDied","Data":"14436c37d3bc4d35e7831a8f5efd028b6a132ddedfb03b78665caa3358cc0b49"} Jul 12 18:43:09.539077 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.538821 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b" event={"ID":"b2953010-656d-4a10-a530-a36ae0b215dc","Type":"ContainerDied","Data":"b9e799fde329db0139ce18494166722810bea9408bd1e8e95f14cb6cf0580488"} Jul 12 18:43:09.539077 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.538846 2576 scope.go:117] "RemoveContainer" containerID="14436c37d3bc4d35e7831a8f5efd028b6a132ddedfb03b78665caa3358cc0b49" Jul 12 18:43:09.548242 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.548224 2576 scope.go:117] "RemoveContainer" containerID="14436c37d3bc4d35e7831a8f5efd028b6a132ddedfb03b78665caa3358cc0b49" Jul 12 18:43:09.548543 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:43:09.548515 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14436c37d3bc4d35e7831a8f5efd028b6a132ddedfb03b78665caa3358cc0b49\": container with ID starting with 14436c37d3bc4d35e7831a8f5efd028b6a132ddedfb03b78665caa3358cc0b49 not found: ID does not exist" containerID="14436c37d3bc4d35e7831a8f5efd028b6a132ddedfb03b78665caa3358cc0b49" Jul 12 18:43:09.548647 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.548551 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14436c37d3bc4d35e7831a8f5efd028b6a132ddedfb03b78665caa3358cc0b49"} err="failed to get container status \"14436c37d3bc4d35e7831a8f5efd028b6a132ddedfb03b78665caa3358cc0b49\": rpc error: code = NotFound desc = could not find container \"14436c37d3bc4d35e7831a8f5efd028b6a132ddedfb03b78665caa3358cc0b49\": container with ID starting with 14436c37d3bc4d35e7831a8f5efd028b6a132ddedfb03b78665caa3358cc0b49 not found: ID does not exist" Jul 12 18:43:09.559303 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.559276 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b"] Jul 12 18:43:09.563329 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:09.563310 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b7e51ce2-957c4fb9b-dh67b"] Jul 12 18:43:10.818608 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:10.818574 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2953010-656d-4a10-a530-a36ae0b215dc" path="/var/lib/kubelet/pods/b2953010-656d-4a10-a530-a36ae0b215dc/volumes" Jul 12 18:43:14.732131 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:14.732096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:14.831059 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:14.831024 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:16.570575 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:16.570542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:20.596778 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:20.596738 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:20.651476 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:20.651443 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:20.886270 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:20.886184 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:20.977066 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:20.977035 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:22.020351 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.020315 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:22.091154 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.091119 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:22.092815 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.092774 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd"] Jul 12 18:43:22.093484 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.093466 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2953010-656d-4a10-a530-a36ae0b215dc" containerName="maas-api" Jul 12 18:43:22.093527 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.093490 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2953010-656d-4a10-a530-a36ae0b215dc" containerName="maas-api" Jul 12 18:43:22.093616 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.093605 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2953010-656d-4a10-a530-a36ae0b215dc" containerName="maas-api" Jul 12 18:43:22.098368 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.098349 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" Jul 12 18:43:22.101931 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.101710 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-93b7e9bc-serving-cert\"" Jul 12 18:43:22.104511 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.104487 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd"] Jul 12 18:43:22.159691 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.159650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:22.233055 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.233014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26e7e715-cb61-4ccc-91ef-6766a6fed798-openshift-service-ca\") pod \"maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd\" (UID: \"26e7e715-cb61-4ccc-91ef-6766a6fed798\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" Jul 12 18:43:22.233245 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.233188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26e7e715-cb61-4ccc-91ef-6766a6fed798-maas-api-tls\") pod \"maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd\" (UID: \"26e7e715-cb61-4ccc-91ef-6766a6fed798\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" Jul 12 18:43:22.233245 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.233217 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll4vb\" (UniqueName: \"kubernetes.io/projected/26e7e715-cb61-4ccc-91ef-6766a6fed798-kube-api-access-ll4vb\") pod \"maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd\" (UID: \"26e7e715-cb61-4ccc-91ef-6766a6fed798\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" Jul 12 18:43:22.334314 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.334279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26e7e715-cb61-4ccc-91ef-6766a6fed798-openshift-service-ca\") pod \"maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd\" (UID: \"26e7e715-cb61-4ccc-91ef-6766a6fed798\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" Jul 12 18:43:22.334516 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.334382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26e7e715-cb61-4ccc-91ef-6766a6fed798-maas-api-tls\") pod \"maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd\" (UID: \"26e7e715-cb61-4ccc-91ef-6766a6fed798\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" Jul 12 18:43:22.334516 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.334403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll4vb\" (UniqueName: \"kubernetes.io/projected/26e7e715-cb61-4ccc-91ef-6766a6fed798-kube-api-access-ll4vb\") pod \"maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd\" (UID: \"26e7e715-cb61-4ccc-91ef-6766a6fed798\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" Jul 12 18:43:22.335020 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.334993 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26e7e715-cb61-4ccc-91ef-6766a6fed798-openshift-service-ca\") pod \"maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd\" (UID: \"26e7e715-cb61-4ccc-91ef-6766a6fed798\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" Jul 12 18:43:22.336898 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.336874 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26e7e715-cb61-4ccc-91ef-6766a6fed798-maas-api-tls\") pod \"maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd\" (UID: \"26e7e715-cb61-4ccc-91ef-6766a6fed798\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" Jul 12 18:43:22.342249 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.342223 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll4vb\" (UniqueName: \"kubernetes.io/projected/26e7e715-cb61-4ccc-91ef-6766a6fed798-kube-api-access-ll4vb\") pod \"maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd\" (UID: \"26e7e715-cb61-4ccc-91ef-6766a6fed798\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" Jul 12 18:43:22.357711 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.357687 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:22.412508 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.412477 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" Jul 12 18:43:22.565648 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.565494 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd"] Jul 12 18:43:22.568108 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:43:22.568075 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e7e715_cb61_4ccc_91ef_6766a6fed798.slice/crio-ffb3426ccee724f34b9054fd9137de36efd32b484318ccf9bdcfae4ed7ebea81 WatchSource:0}: Error finding container ffb3426ccee724f34b9054fd9137de36efd32b484318ccf9bdcfae4ed7ebea81: Status 404 returned error can't find the container with id ffb3426ccee724f34b9054fd9137de36efd32b484318ccf9bdcfae4ed7ebea81 Jul 12 18:43:22.574372 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.574344 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:22.611844 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.611732 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" event={"ID":"26e7e715-cb61-4ccc-91ef-6766a6fed798","Type":"ContainerStarted","Data":"ffb3426ccee724f34b9054fd9137de36efd32b484318ccf9bdcfae4ed7ebea81"} Jul 12 18:43:22.676335 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:22.676303 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:23.618232 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:23.618189 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" event={"ID":"26e7e715-cb61-4ccc-91ef-6766a6fed798","Type":"ContainerStarted","Data":"8c37f13fcf8f8daaef6c223fbceb3458289a804bd56a47fe4c4e617337358464"} Jul 12 18:43:23.618663 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:23.618254 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" Jul 12 18:43:23.635414 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:23.635363 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" podStartSLOduration=1.288213905 podStartE2EDuration="1.635344546s" totalImagesPullingTime="347.130641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:43:23.634143214 +0000 UTC m=+1565.401440798" watchObservedRunningTime="2026-07-12 18:43:23.635344546 +0000 UTC m=+1565.402642127" Jul 12 18:43:24.238313 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:24.238283 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:26.354547 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:26.354508 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:26.550565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:26.550527 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:26.647365 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:26.647277 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:26.764259 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:26.764221 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd"] Jul 12 18:43:26.764460 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:26.764433 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" podUID="26e7e715-cb61-4ccc-91ef-6766a6fed798" containerName="maas-api" containerID="cri-o://8c37f13fcf8f8daaef6c223fbceb3458289a804bd56a47fe4c4e617337358464" gracePeriod=30 Jul 12 18:43:26.789845 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:26.789773 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:27.029612 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.029580 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" Jul 12 18:43:27.178673 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.178644 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26e7e715-cb61-4ccc-91ef-6766a6fed798-openshift-service-ca\") pod \"26e7e715-cb61-4ccc-91ef-6766a6fed798\" (UID: \"26e7e715-cb61-4ccc-91ef-6766a6fed798\") " Jul 12 18:43:27.178673 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.178678 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll4vb\" (UniqueName: \"kubernetes.io/projected/26e7e715-cb61-4ccc-91ef-6766a6fed798-kube-api-access-ll4vb\") pod \"26e7e715-cb61-4ccc-91ef-6766a6fed798\" (UID: \"26e7e715-cb61-4ccc-91ef-6766a6fed798\") " Jul 12 18:43:27.178921 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.178759 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26e7e715-cb61-4ccc-91ef-6766a6fed798-maas-api-tls\") pod \"26e7e715-cb61-4ccc-91ef-6766a6fed798\" (UID: \"26e7e715-cb61-4ccc-91ef-6766a6fed798\") " Jul 12 18:43:27.179070 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.179041 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e7e715-cb61-4ccc-91ef-6766a6fed798-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "26e7e715-cb61-4ccc-91ef-6766a6fed798" (UID: "26e7e715-cb61-4ccc-91ef-6766a6fed798"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:43:27.180870 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.180838 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e7e715-cb61-4ccc-91ef-6766a6fed798-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "26e7e715-cb61-4ccc-91ef-6766a6fed798" (UID: "26e7e715-cb61-4ccc-91ef-6766a6fed798"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:43:27.180991 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.180885 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e7e715-cb61-4ccc-91ef-6766a6fed798-kube-api-access-ll4vb" (OuterVolumeSpecName: "kube-api-access-ll4vb") pod "26e7e715-cb61-4ccc-91ef-6766a6fed798" (UID: "26e7e715-cb61-4ccc-91ef-6766a6fed798"). InnerVolumeSpecName "kube-api-access-ll4vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:43:27.280094 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.279998 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26e7e715-cb61-4ccc-91ef-6766a6fed798-openshift-service-ca\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:43:27.280094 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.280028 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll4vb\" (UniqueName: \"kubernetes.io/projected/26e7e715-cb61-4ccc-91ef-6766a6fed798-kube-api-access-ll4vb\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:43:27.280094 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.280038 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26e7e715-cb61-4ccc-91ef-6766a6fed798-maas-api-tls\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:43:27.640753 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.640719 2576 generic.go:356] "Generic (PLEG): container finished" podID="26e7e715-cb61-4ccc-91ef-6766a6fed798" containerID="8c37f13fcf8f8daaef6c223fbceb3458289a804bd56a47fe4c4e617337358464" exitCode=0 Jul 12 18:43:27.641208 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.640764 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" Jul 12 18:43:27.641208 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.640832 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" event={"ID":"26e7e715-cb61-4ccc-91ef-6766a6fed798","Type":"ContainerDied","Data":"8c37f13fcf8f8daaef6c223fbceb3458289a804bd56a47fe4c4e617337358464"} Jul 12 18:43:27.641208 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.640891 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd" event={"ID":"26e7e715-cb61-4ccc-91ef-6766a6fed798","Type":"ContainerDied","Data":"ffb3426ccee724f34b9054fd9137de36efd32b484318ccf9bdcfae4ed7ebea81"} Jul 12 18:43:27.641208 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.640925 2576 scope.go:117] "RemoveContainer" containerID="8c37f13fcf8f8daaef6c223fbceb3458289a804bd56a47fe4c4e617337358464" Jul 12 18:43:27.650141 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.650123 2576 scope.go:117] "RemoveContainer" containerID="8c37f13fcf8f8daaef6c223fbceb3458289a804bd56a47fe4c4e617337358464" Jul 12 18:43:27.650391 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:43:27.650370 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c37f13fcf8f8daaef6c223fbceb3458289a804bd56a47fe4c4e617337358464\": container with ID starting with 8c37f13fcf8f8daaef6c223fbceb3458289a804bd56a47fe4c4e617337358464 not found: ID does not exist" containerID="8c37f13fcf8f8daaef6c223fbceb3458289a804bd56a47fe4c4e617337358464" Jul 12 18:43:27.650436 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.650403 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c37f13fcf8f8daaef6c223fbceb3458289a804bd56a47fe4c4e617337358464"} err="failed to get container status \"8c37f13fcf8f8daaef6c223fbceb3458289a804bd56a47fe4c4e617337358464\": rpc error: code = NotFound desc = could not find container \"8c37f13fcf8f8daaef6c223fbceb3458289a804bd56a47fe4c4e617337358464\": container with ID starting with 8c37f13fcf8f8daaef6c223fbceb3458289a804bd56a47fe4c4e617337358464 not found: ID does not exist" Jul 12 18:43:27.661911 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.661875 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd"] Jul 12 18:43:27.663690 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:27.663672 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93b7e9bc-796db99f79-82pjd"] Jul 12 18:43:28.166488 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:28.166459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:28.818136 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:28.818102 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26e7e715-cb61-4ccc-91ef-6766a6fed798" path="/var/lib/kubelet/pods/26e7e715-cb61-4ccc-91ef-6766a6fed798/volumes" Jul 12 18:43:32.593877 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:32.593841 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:32.658752 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:32.658716 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:33.079462 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:33.079428 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:33.181298 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:33.181264 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:33.278916 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:33.278881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:34.573699 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:34.573644 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:40.521097 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:40.521046 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:40.571687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:40.571652 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:40.727127 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:40.727085 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:40.832508 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:40.832473 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:43:41.243490 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:43:41.243404 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:07.212773 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:07.212691 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:12.891098 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:12.891022 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:12.998310 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:12.998276 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:13.101240 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:13.101206 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:28.977266 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:28.977233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:34.710140 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:34.710106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:34.778203 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:34.778170 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:34.859547 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:34.859515 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:34.959092 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:34.959051 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:45.131112 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:45.131078 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:45.434844 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:45.434722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:46.193996 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:46.193956 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:52.150524 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:52.150477 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:52.243529 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:52.243491 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:52.345193 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:52.345164 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:52.443427 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:52.443345 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:53.202405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:53.202371 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:53.297533 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:53.297498 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:53.397159 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:53.397122 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:44:54.926781 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:44:54.926747 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:00.138778 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:00.138744 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd"] Jul 12 18:45:00.139187 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:00.139173 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e7e715-cb61-4ccc-91ef-6766a6fed798" containerName="maas-api" Jul 12 18:45:00.139187 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:00.139188 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e7e715-cb61-4ccc-91ef-6766a6fed798" containerName="maas-api" Jul 12 18:45:00.139279 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:00.139258 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e7e715-cb61-4ccc-91ef-6766a6fed798" containerName="maas-api" Jul 12 18:45:00.142498 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:00.142481 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd" Jul 12 18:45:00.150584 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:00.150562 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd"] Jul 12 18:45:00.242532 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:00.242494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9b9\" (UniqueName: \"kubernetes.io/projected/0b7650fa-d9c6-4102-8a57-44bf973a74d2-kube-api-access-dw9b9\") pod \"maas-api-key-cleanup-29731365-ql2cd\" (UID: \"0b7650fa-d9c6-4102-8a57-44bf973a74d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd" Jul 12 18:45:00.343716 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:00.343682 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9b9\" (UniqueName: \"kubernetes.io/projected/0b7650fa-d9c6-4102-8a57-44bf973a74d2-kube-api-access-dw9b9\") pod \"maas-api-key-cleanup-29731365-ql2cd\" (UID: \"0b7650fa-d9c6-4102-8a57-44bf973a74d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd" Jul 12 18:45:00.352070 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:00.352039 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9b9\" (UniqueName: \"kubernetes.io/projected/0b7650fa-d9c6-4102-8a57-44bf973a74d2-kube-api-access-dw9b9\") pod \"maas-api-key-cleanup-29731365-ql2cd\" (UID: \"0b7650fa-d9c6-4102-8a57-44bf973a74d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd" Jul 12 18:45:00.453548 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:00.453459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd" Jul 12 18:45:00.588712 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:00.588678 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd"] Jul 12 18:45:00.590830 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:45:00.590801 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7650fa_d9c6_4102_8a57_44bf973a74d2.slice/crio-d4aa8aefc6d5e74a36df1735b17d8118bb82077c73004afd9ee5f07f781fba17 WatchSource:0}: Error finding container d4aa8aefc6d5e74a36df1735b17d8118bb82077c73004afd9ee5f07f781fba17: Status 404 returned error can't find the container with id d4aa8aefc6d5e74a36df1735b17d8118bb82077c73004afd9ee5f07f781fba17 Jul 12 18:45:00.794272 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:00.794186 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:00.853693 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:00.853661 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:01.124984 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:01.124945 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd" event={"ID":"0b7650fa-d9c6-4102-8a57-44bf973a74d2","Type":"ContainerStarted","Data":"6aa212d86deb3bf8c0ae12b66f70598e75befa5a5342d8ae902c1c3251cfa9c9"} Jul 12 18:45:01.125178 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:01.125006 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd" event={"ID":"0b7650fa-d9c6-4102-8a57-44bf973a74d2","Type":"ContainerStarted","Data":"d4aa8aefc6d5e74a36df1735b17d8118bb82077c73004afd9ee5f07f781fba17"} Jul 12 18:45:01.150706 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:01.150659 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd" podStartSLOduration=1.1506440900000001 podStartE2EDuration="1.15064409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:45:01.149295769 +0000 UTC m=+1662.916593351" watchObservedRunningTime="2026-07-12 18:45:01.15064409 +0000 UTC m=+1662.917941671" Jul 12 18:45:01.462928 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:01.462834 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:07.221957 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:07.221908 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:07.323002 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:07.322971 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:07.456564 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:07.456530 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:07.779214 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:07.779175 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:07.930224 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:07.930190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:10.420827 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:10.420723 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:10.503203 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:10.503160 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw"] Jul 12 18:45:10.504417 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:10.504367 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:10.507771 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:10.507751 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" Jul 12 18:45:10.510281 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:10.510263 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2e12c194-serving-cert\"" Jul 12 18:45:10.520156 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:10.520134 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw"] Jul 12 18:45:10.538989 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:10.538955 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc80008-814c-47b5-ae48-ef10890359b0-openshift-service-ca\") pod \"maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw\" (UID: \"2bc80008-814c-47b5-ae48-ef10890359b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" Jul 12 18:45:10.539144 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:10.539004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg7h7\" (UniqueName: \"kubernetes.io/projected/2bc80008-814c-47b5-ae48-ef10890359b0-kube-api-access-bg7h7\") pod \"maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw\" (UID: \"2bc80008-814c-47b5-ae48-ef10890359b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" Jul 12 18:45:10.539144 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:10.539037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc80008-814c-47b5-ae48-ef10890359b0-maas-api-tls\") pod \"maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw\" (UID: \"2bc80008-814c-47b5-ae48-ef10890359b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" Jul 12 18:45:10.586894 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:10.586860 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:10.640444 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:10.640413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc80008-814c-47b5-ae48-ef10890359b0-openshift-service-ca\") pod \"maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw\" (UID: \"2bc80008-814c-47b5-ae48-ef10890359b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" Jul 12 18:45:10.640620 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:10.640456 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg7h7\" (UniqueName: \"kubernetes.io/projected/2bc80008-814c-47b5-ae48-ef10890359b0-kube-api-access-bg7h7\") pod \"maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw\" (UID: \"2bc80008-814c-47b5-ae48-ef10890359b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" Jul 12 18:45:10.640620 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:10.640515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc80008-814c-47b5-ae48-ef10890359b0-maas-api-tls\") pod \"maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw\" (UID: \"2bc80008-814c-47b5-ae48-ef10890359b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" Jul 12 18:45:10.640693 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:45:10.640630 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-serving-cert: secret "maas-api-e2e-mt-2e12c194-serving-cert" not found Jul 12 18:45:10.640693 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:45:10.640681 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc80008-814c-47b5-ae48-ef10890359b0-maas-api-tls podName:2bc80008-814c-47b5-ae48-ef10890359b0 nodeName:}" failed. No retries permitted until 2026-07-12 18:45:11.140664997 +0000 UTC m=+1672.907962557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2bc80008-814c-47b5-ae48-ef10890359b0-maas-api-tls") pod "maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" (UID: "2bc80008-814c-47b5-ae48-ef10890359b0") : secret "maas-api-e2e-mt-2e12c194-serving-cert" not found Jul 12 18:45:10.641251 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:10.641233 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc80008-814c-47b5-ae48-ef10890359b0-openshift-service-ca\") pod \"maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw\" (UID: \"2bc80008-814c-47b5-ae48-ef10890359b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" Jul 12 18:45:10.651914 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:10.651887 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg7h7\" (UniqueName: \"kubernetes.io/projected/2bc80008-814c-47b5-ae48-ef10890359b0-kube-api-access-bg7h7\") pod \"maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw\" (UID: \"2bc80008-814c-47b5-ae48-ef10890359b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" Jul 12 18:45:10.840202 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:10.840120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:11.102421 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:11.102326 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:11.146143 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:11.146108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc80008-814c-47b5-ae48-ef10890359b0-maas-api-tls\") pod \"maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw\" (UID: \"2bc80008-814c-47b5-ae48-ef10890359b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" Jul 12 18:45:11.148494 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:11.148463 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc80008-814c-47b5-ae48-ef10890359b0-maas-api-tls\") pod \"maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw\" (UID: \"2bc80008-814c-47b5-ae48-ef10890359b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" Jul 12 18:45:11.203396 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:11.203355 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:11.421391 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:11.421305 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" Jul 12 18:45:11.564954 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:11.564863 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw"] Jul 12 18:45:11.567456 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:45:11.567430 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc80008_814c_47b5_ae48_ef10890359b0.slice/crio-b1f32201055dfb3a402a6e5526b40f72dc9a440887feb83f6a0bf2b1e209e3da WatchSource:0}: Error finding container b1f32201055dfb3a402a6e5526b40f72dc9a440887feb83f6a0bf2b1e209e3da: Status 404 returned error can't find the container with id b1f32201055dfb3a402a6e5526b40f72dc9a440887feb83f6a0bf2b1e209e3da Jul 12 18:45:12.183382 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:12.183339 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" event={"ID":"2bc80008-814c-47b5-ae48-ef10890359b0","Type":"ContainerStarted","Data":"a73d037d2cb518aece2f4580b6e996c632d34c01ee32762d9b2e9fbce73b4f7a"} Jul 12 18:45:12.183567 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:12.183405 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" event={"ID":"2bc80008-814c-47b5-ae48-ef10890359b0","Type":"ContainerStarted","Data":"b1f32201055dfb3a402a6e5526b40f72dc9a440887feb83f6a0bf2b1e209e3da"} Jul 12 18:45:12.183567 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:12.183547 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" Jul 12 18:45:12.202506 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:12.202460 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" podStartSLOduration=1.755635826 podStartE2EDuration="2.202442518s" totalImagesPullingTime="446.806692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:45:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:45:12.200376349 +0000 UTC m=+1673.967673928" watchObservedRunningTime="2026-07-12 18:45:12.202442518 +0000 UTC m=+1673.969740101" Jul 12 18:45:14.905567 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:14.905533 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:15.545845 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:15.545781 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:15.645138 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:15.645102 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:15.950319 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:15.950285 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:18.196104 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:18.196073 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" Jul 12 18:45:21.495642 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:21.495608 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:21.550821 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:21.550776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:21.881111 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:21.881072 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:22.175928 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:22.175834 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw"] Jul 12 18:45:22.176578 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:22.176547 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" podUID="2bc80008-814c-47b5-ae48-ef10890359b0" containerName="maas-api" containerID="cri-o://a73d037d2cb518aece2f4580b6e996c632d34c01ee32762d9b2e9fbce73b4f7a" gracePeriod=30 Jul 12 18:45:22.199606 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:22.199571 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:22.201948 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:45:22.201927 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc80008_814c_47b5_ae48_ef10890359b0.slice/crio-a73d037d2cb518aece2f4580b6e996c632d34c01ee32762d9b2e9fbce73b4f7a.scope\": RecentStats: unable to find data in memory cache]" Jul 12 18:45:22.271317 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:22.271286 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:22.430046 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:22.429988 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" Jul 12 18:45:22.556806 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:22.556747 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc80008-814c-47b5-ae48-ef10890359b0-maas-api-tls\") pod \"2bc80008-814c-47b5-ae48-ef10890359b0\" (UID: \"2bc80008-814c-47b5-ae48-ef10890359b0\") " Jul 12 18:45:22.557246 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:22.556871 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc80008-814c-47b5-ae48-ef10890359b0-openshift-service-ca\") pod \"2bc80008-814c-47b5-ae48-ef10890359b0\" (UID: \"2bc80008-814c-47b5-ae48-ef10890359b0\") " Jul 12 18:45:22.557246 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:22.556940 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bg7h7\" (UniqueName: \"kubernetes.io/projected/2bc80008-814c-47b5-ae48-ef10890359b0-kube-api-access-bg7h7\") pod \"2bc80008-814c-47b5-ae48-ef10890359b0\" (UID: \"2bc80008-814c-47b5-ae48-ef10890359b0\") " Jul 12 18:45:22.557246 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:22.557209 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc80008-814c-47b5-ae48-ef10890359b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2bc80008-814c-47b5-ae48-ef10890359b0" (UID: "2bc80008-814c-47b5-ae48-ef10890359b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:45:22.558933 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:22.558911 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc80008-814c-47b5-ae48-ef10890359b0-kube-api-access-bg7h7" (OuterVolumeSpecName: "kube-api-access-bg7h7") pod "2bc80008-814c-47b5-ae48-ef10890359b0" (UID: "2bc80008-814c-47b5-ae48-ef10890359b0"). InnerVolumeSpecName "kube-api-access-bg7h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:45:22.558933 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:22.558923 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bc80008-814c-47b5-ae48-ef10890359b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2bc80008-814c-47b5-ae48-ef10890359b0" (UID: "2bc80008-814c-47b5-ae48-ef10890359b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:45:22.658040 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:22.658004 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc80008-814c-47b5-ae48-ef10890359b0-openshift-service-ca\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:45:22.658040 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:22.658035 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg7h7\" (UniqueName: \"kubernetes.io/projected/2bc80008-814c-47b5-ae48-ef10890359b0-kube-api-access-bg7h7\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:45:22.658040 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:22.658045 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc80008-814c-47b5-ae48-ef10890359b0-maas-api-tls\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:45:23.248778 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:23.248682 2576 generic.go:356] "Generic (PLEG): container finished" podID="2bc80008-814c-47b5-ae48-ef10890359b0" containerID="a73d037d2cb518aece2f4580b6e996c632d34c01ee32762d9b2e9fbce73b4f7a" exitCode=0 Jul 12 18:45:23.248778 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:23.248728 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" Jul 12 18:45:23.248997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:23.248772 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" event={"ID":"2bc80008-814c-47b5-ae48-ef10890359b0","Type":"ContainerDied","Data":"a73d037d2cb518aece2f4580b6e996c632d34c01ee32762d9b2e9fbce73b4f7a"} Jul 12 18:45:23.248997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:23.248854 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw" event={"ID":"2bc80008-814c-47b5-ae48-ef10890359b0","Type":"ContainerDied","Data":"b1f32201055dfb3a402a6e5526b40f72dc9a440887feb83f6a0bf2b1e209e3da"} Jul 12 18:45:23.248997 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:23.248890 2576 scope.go:117] "RemoveContainer" containerID="a73d037d2cb518aece2f4580b6e996c632d34c01ee32762d9b2e9fbce73b4f7a" Jul 12 18:45:23.257931 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:23.257911 2576 scope.go:117] "RemoveContainer" containerID="a73d037d2cb518aece2f4580b6e996c632d34c01ee32762d9b2e9fbce73b4f7a" Jul 12 18:45:23.258211 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:45:23.258192 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73d037d2cb518aece2f4580b6e996c632d34c01ee32762d9b2e9fbce73b4f7a\": container with ID starting with a73d037d2cb518aece2f4580b6e996c632d34c01ee32762d9b2e9fbce73b4f7a not found: ID does not exist" containerID="a73d037d2cb518aece2f4580b6e996c632d34c01ee32762d9b2e9fbce73b4f7a" Jul 12 18:45:23.258263 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:23.258220 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73d037d2cb518aece2f4580b6e996c632d34c01ee32762d9b2e9fbce73b4f7a"} err="failed to get container status \"a73d037d2cb518aece2f4580b6e996c632d34c01ee32762d9b2e9fbce73b4f7a\": rpc error: code = NotFound desc = could not find container \"a73d037d2cb518aece2f4580b6e996c632d34c01ee32762d9b2e9fbce73b4f7a\": container with ID starting with a73d037d2cb518aece2f4580b6e996c632d34c01ee32762d9b2e9fbce73b4f7a not found: ID does not exist" Jul 12 18:45:23.264516 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:23.264495 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw"] Jul 12 18:45:23.270539 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:23.270517 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2e12c194-748b7fbd8b-d6stw"] Jul 12 18:45:23.649600 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:23.649563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:24.817959 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:24.817922 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bc80008-814c-47b5-ae48-ef10890359b0" path="/var/lib/kubelet/pods/2bc80008-814c-47b5-ae48-ef10890359b0/volumes" Jul 12 18:45:27.917240 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:27.917207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:28.038725 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:28.038692 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:28.205429 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:28.205249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:28.532321 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:28.532207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:28.633721 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:28.633687 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:28.730464 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:28.730426 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:34.433562 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:34.433525 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:34.493427 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:34.493393 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:34.564723 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:34.564682 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:34.661429 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:34.661386 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:34.777668 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:34.777578 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:40.642538 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:40.642456 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:40.697257 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:40.697215 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:47.369339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:47.369294 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:47.661140 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:47.661045 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:47.785054 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:47.785005 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:50.528034 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.527997 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:50.627089 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.627055 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:50.697388 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.697356 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l"] Jul 12 18:45:50.697797 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.697773 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bc80008-814c-47b5-ae48-ef10890359b0" containerName="maas-api" Jul 12 18:45:50.697848 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.697799 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bc80008-814c-47b5-ae48-ef10890359b0" containerName="maas-api" Jul 12 18:45:50.697888 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.697866 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bc80008-814c-47b5-ae48-ef10890359b0" containerName="maas-api" Jul 12 18:45:50.701000 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.700985 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" Jul 12 18:45:50.703415 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.703393 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0328d158-serving-cert\"" Jul 12 18:45:50.715563 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.715528 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l"] Jul 12 18:45:50.808322 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.808236 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77db450c-63dc-4a28-b462-f0ba60ad3e4e-openshift-service-ca\") pod \"maas-api-e2e-mt-0328d158-7cc658db4d-6k98l\" (UID: \"77db450c-63dc-4a28-b462-f0ba60ad3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" Jul 12 18:45:50.808322 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.808289 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65b5j\" (UniqueName: \"kubernetes.io/projected/77db450c-63dc-4a28-b462-f0ba60ad3e4e-kube-api-access-65b5j\") pod \"maas-api-e2e-mt-0328d158-7cc658db4d-6k98l\" (UID: \"77db450c-63dc-4a28-b462-f0ba60ad3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" Jul 12 18:45:50.808503 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.808400 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77db450c-63dc-4a28-b462-f0ba60ad3e4e-maas-api-tls\") pod \"maas-api-e2e-mt-0328d158-7cc658db4d-6k98l\" (UID: \"77db450c-63dc-4a28-b462-f0ba60ad3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" Jul 12 18:45:50.848060 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.848027 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:50.908946 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.908908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65b5j\" (UniqueName: \"kubernetes.io/projected/77db450c-63dc-4a28-b462-f0ba60ad3e4e-kube-api-access-65b5j\") pod \"maas-api-e2e-mt-0328d158-7cc658db4d-6k98l\" (UID: \"77db450c-63dc-4a28-b462-f0ba60ad3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" Jul 12 18:45:50.909143 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.908961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77db450c-63dc-4a28-b462-f0ba60ad3e4e-maas-api-tls\") pod \"maas-api-e2e-mt-0328d158-7cc658db4d-6k98l\" (UID: \"77db450c-63dc-4a28-b462-f0ba60ad3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" Jul 12 18:45:50.909143 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.909022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77db450c-63dc-4a28-b462-f0ba60ad3e4e-openshift-service-ca\") pod \"maas-api-e2e-mt-0328d158-7cc658db4d-6k98l\" (UID: \"77db450c-63dc-4a28-b462-f0ba60ad3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" Jul 12 18:45:50.909813 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.909770 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77db450c-63dc-4a28-b462-f0ba60ad3e4e-openshift-service-ca\") pod \"maas-api-e2e-mt-0328d158-7cc658db4d-6k98l\" (UID: \"77db450c-63dc-4a28-b462-f0ba60ad3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" Jul 12 18:45:50.911389 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.911371 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77db450c-63dc-4a28-b462-f0ba60ad3e4e-maas-api-tls\") pod \"maas-api-e2e-mt-0328d158-7cc658db4d-6k98l\" (UID: \"77db450c-63dc-4a28-b462-f0ba60ad3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" Jul 12 18:45:50.918518 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.918494 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65b5j\" (UniqueName: \"kubernetes.io/projected/77db450c-63dc-4a28-b462-f0ba60ad3e4e-kube-api-access-65b5j\") pod \"maas-api-e2e-mt-0328d158-7cc658db4d-6k98l\" (UID: \"77db450c-63dc-4a28-b462-f0ba60ad3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" Jul 12 18:45:50.947150 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:50.947118 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:51.011928 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:51.011900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" Jul 12 18:45:51.154866 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:51.154840 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l"] Jul 12 18:45:51.156837 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:45:51.156803 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77db450c_63dc_4a28_b462_f0ba60ad3e4e.slice/crio-777fe0c33ba8c7a23b31543608fc12a9f4fbb578fc01b923623b088c5c314333 WatchSource:0}: Error finding container 777fe0c33ba8c7a23b31543608fc12a9f4fbb578fc01b923623b088c5c314333: Status 404 returned error can't find the container with id 777fe0c33ba8c7a23b31543608fc12a9f4fbb578fc01b923623b088c5c314333 Jul 12 18:45:51.256125 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:51.256086 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:51.356279 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:51.356239 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:51.393823 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:51.393771 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" event={"ID":"77db450c-63dc-4a28-b462-f0ba60ad3e4e","Type":"ContainerStarted","Data":"777fe0c33ba8c7a23b31543608fc12a9f4fbb578fc01b923623b088c5c314333"} Jul 12 18:45:52.400506 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:52.400472 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" event={"ID":"77db450c-63dc-4a28-b462-f0ba60ad3e4e","Type":"ContainerStarted","Data":"b78c057a9c92620820292f964f8c68bf000e85640770b1efdf433079c3182bac"} Jul 12 18:45:52.400898 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:52.400543 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" Jul 12 18:45:52.417618 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:52.417564 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" podStartSLOduration=2.084394501 podStartE2EDuration="2.417549844s" totalImagesPullingTime="333.155343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:45:52.415939855 +0000 UTC m=+1714.183237450" watchObservedRunningTime="2026-07-12 18:45:52.417549844 +0000 UTC m=+1714.184847466" Jul 12 18:45:55.000343 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:55.000304 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:58.411101 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:58.411073 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" Jul 12 18:45:59.370664 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:59.370624 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:45:59.471455 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:45:59.471424 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:46:00.011068 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:00.011038 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd"] Jul 12 18:46:00.011238 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:00.011219 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd" podUID="0b7650fa-d9c6-4102-8a57-44bf973a74d2" containerName="cleanup" containerID="cri-o://6aa212d86deb3bf8c0ae12b66f70598e75befa5a5342d8ae902c1c3251cfa9c9" gracePeriod=30 Jul 12 18:46:03.887660 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:03.887627 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:46:12.124850 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:12.124781 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:46:12.226085 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:12.226052 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:46:27.367800 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:27.367751 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:46:27.666237 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:27.666153 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l"] Jul 12 18:46:27.666452 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:27.666425 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" podUID="77db450c-63dc-4a28-b462-f0ba60ad3e4e" containerName="maas-api" containerID="cri-o://b78c057a9c92620820292f964f8c68bf000e85640770b1efdf433079c3182bac" gracePeriod=30 Jul 12 18:46:27.692123 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:27.692096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:46:27.916068 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:27.916045 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" Jul 12 18:46:27.971168 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:27.971083 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65b5j\" (UniqueName: \"kubernetes.io/projected/77db450c-63dc-4a28-b462-f0ba60ad3e4e-kube-api-access-65b5j\") pod \"77db450c-63dc-4a28-b462-f0ba60ad3e4e\" (UID: \"77db450c-63dc-4a28-b462-f0ba60ad3e4e\") " Jul 12 18:46:27.971168 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:27.971141 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77db450c-63dc-4a28-b462-f0ba60ad3e4e-openshift-service-ca\") pod \"77db450c-63dc-4a28-b462-f0ba60ad3e4e\" (UID: \"77db450c-63dc-4a28-b462-f0ba60ad3e4e\") " Jul 12 18:46:27.971378 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:27.971182 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77db450c-63dc-4a28-b462-f0ba60ad3e4e-maas-api-tls\") pod \"77db450c-63dc-4a28-b462-f0ba60ad3e4e\" (UID: \"77db450c-63dc-4a28-b462-f0ba60ad3e4e\") " Jul 12 18:46:27.971537 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:27.971512 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77db450c-63dc-4a28-b462-f0ba60ad3e4e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "77db450c-63dc-4a28-b462-f0ba60ad3e4e" (UID: "77db450c-63dc-4a28-b462-f0ba60ad3e4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:46:27.973422 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:27.973395 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77db450c-63dc-4a28-b462-f0ba60ad3e4e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "77db450c-63dc-4a28-b462-f0ba60ad3e4e" (UID: "77db450c-63dc-4a28-b462-f0ba60ad3e4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:46:27.973866 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:27.973840 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77db450c-63dc-4a28-b462-f0ba60ad3e4e-kube-api-access-65b5j" (OuterVolumeSpecName: "kube-api-access-65b5j") pod "77db450c-63dc-4a28-b462-f0ba60ad3e4e" (UID: "77db450c-63dc-4a28-b462-f0ba60ad3e4e"). InnerVolumeSpecName "kube-api-access-65b5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:46:28.072546 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:28.072513 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65b5j\" (UniqueName: \"kubernetes.io/projected/77db450c-63dc-4a28-b462-f0ba60ad3e4e-kube-api-access-65b5j\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:46:28.072546 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:28.072539 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77db450c-63dc-4a28-b462-f0ba60ad3e4e-openshift-service-ca\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:46:28.072546 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:28.072549 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77db450c-63dc-4a28-b462-f0ba60ad3e4e-maas-api-tls\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:46:28.596463 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:28.596424 2576 generic.go:356] "Generic (PLEG): container finished" podID="77db450c-63dc-4a28-b462-f0ba60ad3e4e" containerID="b78c057a9c92620820292f964f8c68bf000e85640770b1efdf433079c3182bac" exitCode=0 Jul 12 18:46:28.596930 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:28.596474 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" Jul 12 18:46:28.596930 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:28.596474 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" event={"ID":"77db450c-63dc-4a28-b462-f0ba60ad3e4e","Type":"ContainerDied","Data":"b78c057a9c92620820292f964f8c68bf000e85640770b1efdf433079c3182bac"} Jul 12 18:46:28.596930 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:28.596527 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l" event={"ID":"77db450c-63dc-4a28-b462-f0ba60ad3e4e","Type":"ContainerDied","Data":"777fe0c33ba8c7a23b31543608fc12a9f4fbb578fc01b923623b088c5c314333"} Jul 12 18:46:28.596930 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:28.596559 2576 scope.go:117] "RemoveContainer" containerID="b78c057a9c92620820292f964f8c68bf000e85640770b1efdf433079c3182bac" Jul 12 18:46:28.606223 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:28.606201 2576 scope.go:117] "RemoveContainer" containerID="b78c057a9c92620820292f964f8c68bf000e85640770b1efdf433079c3182bac" Jul 12 18:46:28.606498 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:46:28.606476 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78c057a9c92620820292f964f8c68bf000e85640770b1efdf433079c3182bac\": container with ID starting with b78c057a9c92620820292f964f8c68bf000e85640770b1efdf433079c3182bac not found: ID does not exist" containerID="b78c057a9c92620820292f964f8c68bf000e85640770b1efdf433079c3182bac" Jul 12 18:46:28.606565 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:28.606505 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78c057a9c92620820292f964f8c68bf000e85640770b1efdf433079c3182bac"} err="failed to get container status \"b78c057a9c92620820292f964f8c68bf000e85640770b1efdf433079c3182bac\": rpc error: code = NotFound desc = could not find container \"b78c057a9c92620820292f964f8c68bf000e85640770b1efdf433079c3182bac\": container with ID starting with b78c057a9c92620820292f964f8c68bf000e85640770b1efdf433079c3182bac not found: ID does not exist" Jul 12 18:46:28.619176 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:28.619150 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l"] Jul 12 18:46:28.623484 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:28.623461 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0328d158-7cc658db4d-6k98l"] Jul 12 18:46:28.819007 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:28.818975 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77db450c-63dc-4a28-b462-f0ba60ad3e4e" path="/var/lib/kubelet/pods/77db450c-63dc-4a28-b462-f0ba60ad3e4e/volumes" Jul 12 18:46:30.199606 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:30.199574 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd" Jul 12 18:46:30.292964 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:30.292876 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw9b9\" (UniqueName: \"kubernetes.io/projected/0b7650fa-d9c6-4102-8a57-44bf973a74d2-kube-api-access-dw9b9\") pod \"0b7650fa-d9c6-4102-8a57-44bf973a74d2\" (UID: \"0b7650fa-d9c6-4102-8a57-44bf973a74d2\") " Jul 12 18:46:30.294959 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:30.294932 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7650fa-d9c6-4102-8a57-44bf973a74d2-kube-api-access-dw9b9" (OuterVolumeSpecName: "kube-api-access-dw9b9") pod "0b7650fa-d9c6-4102-8a57-44bf973a74d2" (UID: "0b7650fa-d9c6-4102-8a57-44bf973a74d2"). InnerVolumeSpecName "kube-api-access-dw9b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:46:30.393653 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:30.393618 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw9b9\" (UniqueName: \"kubernetes.io/projected/0b7650fa-d9c6-4102-8a57-44bf973a74d2-kube-api-access-dw9b9\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:46:30.609018 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:30.608981 2576 generic.go:356] "Generic (PLEG): container finished" podID="0b7650fa-d9c6-4102-8a57-44bf973a74d2" containerID="6aa212d86deb3bf8c0ae12b66f70598e75befa5a5342d8ae902c1c3251cfa9c9" exitCode=137 Jul 12 18:46:30.609192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:30.609030 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd" Jul 12 18:46:30.609192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:30.609060 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd" event={"ID":"0b7650fa-d9c6-4102-8a57-44bf973a74d2","Type":"ContainerDied","Data":"6aa212d86deb3bf8c0ae12b66f70598e75befa5a5342d8ae902c1c3251cfa9c9"} Jul 12 18:46:30.609192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:30.609115 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd" event={"ID":"0b7650fa-d9c6-4102-8a57-44bf973a74d2","Type":"ContainerDied","Data":"d4aa8aefc6d5e74a36df1735b17d8118bb82077c73004afd9ee5f07f781fba17"} Jul 12 18:46:30.609192 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:30.609138 2576 scope.go:117] "RemoveContainer" containerID="6aa212d86deb3bf8c0ae12b66f70598e75befa5a5342d8ae902c1c3251cfa9c9" Jul 12 18:46:30.620047 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:30.620007 2576 scope.go:117] "RemoveContainer" containerID="6aa212d86deb3bf8c0ae12b66f70598e75befa5a5342d8ae902c1c3251cfa9c9" Jul 12 18:46:30.620379 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:46:30.620351 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa212d86deb3bf8c0ae12b66f70598e75befa5a5342d8ae902c1c3251cfa9c9\": container with ID starting with 6aa212d86deb3bf8c0ae12b66f70598e75befa5a5342d8ae902c1c3251cfa9c9 not found: ID does not exist" containerID="6aa212d86deb3bf8c0ae12b66f70598e75befa5a5342d8ae902c1c3251cfa9c9" Jul 12 18:46:30.620450 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:30.620388 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa212d86deb3bf8c0ae12b66f70598e75befa5a5342d8ae902c1c3251cfa9c9"} err="failed to get container status \"6aa212d86deb3bf8c0ae12b66f70598e75befa5a5342d8ae902c1c3251cfa9c9\": rpc error: code = NotFound desc = could not find container \"6aa212d86deb3bf8c0ae12b66f70598e75befa5a5342d8ae902c1c3251cfa9c9\": container with ID starting with 6aa212d86deb3bf8c0ae12b66f70598e75befa5a5342d8ae902c1c3251cfa9c9 not found: ID does not exist" Jul 12 18:46:30.634125 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:30.634097 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd"] Jul 12 18:46:30.642317 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:30.642290 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731365-ql2cd"] Jul 12 18:46:30.818170 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:30.818141 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b7650fa-d9c6-4102-8a57-44bf973a74d2" path="/var/lib/kubelet/pods/0b7650fa-d9c6-4102-8a57-44bf973a74d2/volumes" Jul 12 18:46:33.433647 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:33.433603 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:46:38.520186 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:46:38.520153 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:15.224972 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:15.224928 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:15.333728 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:15.333687 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:20.804597 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:20.804565 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:20.860348 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:20.860305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:21.103147 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:21.103115 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:22.370455 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:22.370419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:22.466880 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:22.466841 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:22.564597 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:22.564566 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:23.259018 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:23.258985 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:23.362741 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:23.362703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:23.460901 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:23.460868 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:24.805307 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:24.805273 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:30.842072 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:30.842039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:30.942373 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:30.942332 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:31.630801 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:31.630740 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:37.553214 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:37.553172 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:37.619163 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:37.619129 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:37.702415 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:37.702374 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:37.799318 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:37.799275 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:48.253999 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:48.253963 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:47:48.595319 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:47:48.595273 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:09.562236 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:09.562204 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:15.357003 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:15.356970 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:15.412858 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:15.412828 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:15.774133 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:15.774049 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:16.026264 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:16.026155 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:16.517931 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:16.517885 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:19.421727 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.421691 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:19.516073 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.516039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:19.596279 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.596243 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5"] Jul 12 18:48:19.596924 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.596906 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b7650fa-d9c6-4102-8a57-44bf973a74d2" containerName="cleanup" Jul 12 18:48:19.596982 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.596928 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7650fa-d9c6-4102-8a57-44bf973a74d2" containerName="cleanup" Jul 12 18:48:19.596982 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.596943 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77db450c-63dc-4a28-b462-f0ba60ad3e4e" containerName="maas-api" Jul 12 18:48:19.596982 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.596952 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="77db450c-63dc-4a28-b462-f0ba60ad3e4e" containerName="maas-api" Jul 12 18:48:19.597095 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.597082 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b7650fa-d9c6-4102-8a57-44bf973a74d2" containerName="cleanup" Jul 12 18:48:19.597130 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.597101 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="77db450c-63dc-4a28-b462-f0ba60ad3e4e" containerName="maas-api" Jul 12 18:48:19.600813 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.600772 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" Jul 12 18:48:19.603200 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.603179 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-14e82f-serving-cert\"" Jul 12 18:48:19.607970 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.607945 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5"] Jul 12 18:48:19.636800 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.636749 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:19.727524 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.727425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3175eb4-b4dc-48d5-831f-b8e09325dccd-maas-api-tls\") pod \"maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5\" (UID: \"f3175eb4-b4dc-48d5-831f-b8e09325dccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" Jul 12 18:48:19.727524 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.727486 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvzqm\" (UniqueName: \"kubernetes.io/projected/f3175eb4-b4dc-48d5-831f-b8e09325dccd-kube-api-access-qvzqm\") pod \"maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5\" (UID: \"f3175eb4-b4dc-48d5-831f-b8e09325dccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" Jul 12 18:48:19.727731 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.727659 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3175eb4-b4dc-48d5-831f-b8e09325dccd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5\" (UID: \"f3175eb4-b4dc-48d5-831f-b8e09325dccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" Jul 12 18:48:19.828188 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.828146 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3175eb4-b4dc-48d5-831f-b8e09325dccd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5\" (UID: \"f3175eb4-b4dc-48d5-831f-b8e09325dccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" Jul 12 18:48:19.828369 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.828210 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3175eb4-b4dc-48d5-831f-b8e09325dccd-maas-api-tls\") pod \"maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5\" (UID: \"f3175eb4-b4dc-48d5-831f-b8e09325dccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" Jul 12 18:48:19.828369 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.828242 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvzqm\" (UniqueName: \"kubernetes.io/projected/f3175eb4-b4dc-48d5-831f-b8e09325dccd-kube-api-access-qvzqm\") pod \"maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5\" (UID: \"f3175eb4-b4dc-48d5-831f-b8e09325dccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" Jul 12 18:48:19.828443 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:48:19.828394 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-serving-cert: secret "maas-api-e2e-inf-a-14e82f-serving-cert" not found Jul 12 18:48:19.828477 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:48:19.828460 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3175eb4-b4dc-48d5-831f-b8e09325dccd-maas-api-tls podName:f3175eb4-b4dc-48d5-831f-b8e09325dccd nodeName:}" failed. No retries permitted until 2026-07-12 18:48:20.328440207 +0000 UTC m=+1862.095737781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3175eb4-b4dc-48d5-831f-b8e09325dccd-maas-api-tls") pod "maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" (UID: "f3175eb4-b4dc-48d5-831f-b8e09325dccd") : secret "maas-api-e2e-inf-a-14e82f-serving-cert" not found Jul 12 18:48:19.829006 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.828984 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3175eb4-b4dc-48d5-831f-b8e09325dccd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5\" (UID: \"f3175eb4-b4dc-48d5-831f-b8e09325dccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" Jul 12 18:48:19.837174 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.837150 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvzqm\" (UniqueName: \"kubernetes.io/projected/f3175eb4-b4dc-48d5-831f-b8e09325dccd-kube-api-access-qvzqm\") pod \"maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5\" (UID: \"f3175eb4-b4dc-48d5-831f-b8e09325dccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" Jul 12 18:48:19.989340 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:19.989255 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:20.089553 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:20.089516 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:20.190733 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:20.190701 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:20.333407 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:20.333370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3175eb4-b4dc-48d5-831f-b8e09325dccd-maas-api-tls\") pod \"maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5\" (UID: \"f3175eb4-b4dc-48d5-831f-b8e09325dccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" Jul 12 18:48:20.335852 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:20.335828 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3175eb4-b4dc-48d5-831f-b8e09325dccd-maas-api-tls\") pod \"maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5\" (UID: \"f3175eb4-b4dc-48d5-831f-b8e09325dccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" Jul 12 18:48:20.513438 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:20.513399 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" Jul 12 18:48:20.654534 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:20.654509 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5"] Jul 12 18:48:20.657507 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:48:20.657478 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3175eb4_b4dc_48d5_831f_b8e09325dccd.slice/crio-ab54ba8cc13f4adad77f364360c53937ad27ef9501e20fcd47c75bd91a8863a6 WatchSource:0}: Error finding container ab54ba8cc13f4adad77f364360c53937ad27ef9501e20fcd47c75bd91a8863a6: Status 404 returned error can't find the container with id ab54ba8cc13f4adad77f364360c53937ad27ef9501e20fcd47c75bd91a8863a6 Jul 12 18:48:20.659382 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:20.659361 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 18:48:21.179066 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:21.179029 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" event={"ID":"f3175eb4-b4dc-48d5-831f-b8e09325dccd","Type":"ContainerStarted","Data":"129f458f7a24bd204e280e4cfaf088e152425299fea040db1fa1d147badb3b46"} Jul 12 18:48:21.179262 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:21.179089 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" event={"ID":"f3175eb4-b4dc-48d5-831f-b8e09325dccd","Type":"ContainerStarted","Data":"ab54ba8cc13f4adad77f364360c53937ad27ef9501e20fcd47c75bd91a8863a6"} Jul 12 18:48:21.179262 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:21.179157 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" Jul 12 18:48:21.198465 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:21.198420 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" podStartSLOduration=1.78214078 podStartE2EDuration="2.198404422s" totalImagesPullingTime="416.263642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:48:21.196448335 +0000 UTC m=+1862.963745919" watchObservedRunningTime="2026-07-12 18:48:21.198404422 +0000 UTC m=+1862.965702004" Jul 12 18:48:23.812104 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:23.812065 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:24.141765 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.141732 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:24.414378 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.414300 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv"] Jul 12 18:48:24.418157 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.418131 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.420911 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.420889 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-89494f-openshift-default-dockercfg-zx4t9\"" Jul 12 18:48:24.429655 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.429622 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv"] Jul 12 18:48:24.445168 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.445141 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:24.471590 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.471554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-data\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.471755 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.471602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-credential-socket\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.471755 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.471644 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-envoy\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.471755 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.471693 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-token\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.471755 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.471716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04bd126-ba33-4119-858e-3250ed60f0d0-istiod-ca-cert\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.471755 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.471735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-workload-socket\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.471992 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.471778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxrcx\" (UniqueName: \"kubernetes.io/projected/b04bd126-ba33-4119-858e-3250ed60f0d0-kube-api-access-sxrcx\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.471992 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.471872 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-workload-certs\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.471992 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.471904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-podinfo\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.573171 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.573138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-token\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.573364 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.573179 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04bd126-ba33-4119-858e-3250ed60f0d0-istiod-ca-cert\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.573364 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.573203 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-workload-socket\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.573364 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.573236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxrcx\" (UniqueName: \"kubernetes.io/projected/b04bd126-ba33-4119-858e-3250ed60f0d0-kube-api-access-sxrcx\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.573364 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.573268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-workload-certs\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.573364 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.573290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-podinfo\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.573364 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.573351 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-data\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.573686 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.573392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-credential-socket\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.573686 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.573429 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-envoy\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.573686 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.573640 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-workload-socket\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.573887 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.573691 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-workload-certs\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.573887 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.573769 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-credential-socket\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.573887 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.573858 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-data\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.573996 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.573980 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04bd126-ba33-4119-858e-3250ed60f0d0-istiod-ca-cert\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.575604 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.575585 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-envoy\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.575870 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.575845 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-podinfo\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.580901 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.580878 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxrcx\" (UniqueName: \"kubernetes.io/projected/b04bd126-ba33-4119-858e-3250ed60f0d0-kube-api-access-sxrcx\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.580901 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.580888 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-token\") pod \"e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.730707 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.730611 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:24.885511 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.885396 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv"] Jul 12 18:48:24.888309 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:48:24.888279 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04bd126_ba33_4119_858e_3250ed60f0d0.slice/crio-3bd69b9ae506639a0a696859ca9408a6a5b50e6d22330f0405d159ebeb524773 WatchSource:0}: Error finding container 3bd69b9ae506639a0a696859ca9408a6a5b50e6d22330f0405d159ebeb524773: Status 404 returned error can't find the container with id 3bd69b9ae506639a0a696859ca9408a6a5b50e6d22330f0405d159ebeb524773 Jul 12 18:48:24.890719 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.890684 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:48:24.890833 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.890767 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:48:24.890906 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.890829 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:48:24.923864 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:24.923835 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:25.202779 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:25.202743 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" event={"ID":"b04bd126-ba33-4119-858e-3250ed60f0d0","Type":"ContainerStarted","Data":"28f70be01239131f28dc78eea92f3aacff4bd8104cc8172d03e4f30247cc3223"} Jul 12 18:48:25.202973 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:25.202815 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" event={"ID":"b04bd126-ba33-4119-858e-3250ed60f0d0","Type":"ContainerStarted","Data":"3bd69b9ae506639a0a696859ca9408a6a5b50e6d22330f0405d159ebeb524773"} Jul 12 18:48:25.224061 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:25.224016 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" podStartSLOduration=1.22400284 podStartE2EDuration="1.22400284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:48:25.222622726 +0000 UTC m=+1866.989920335" watchObservedRunningTime="2026-07-12 18:48:25.22400284 +0000 UTC m=+1866.991300421" Jul 12 18:48:25.731736 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:25.731700 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:25.737264 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:25.737240 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:26.208298 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:26.208266 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:26.209286 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:26.209263 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:27.107890 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.107855 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:27.178621 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.178584 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8"] Jul 12 18:48:27.180324 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.180291 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:27.183111 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.183091 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" Jul 12 18:48:27.185529 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.185512 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-89494f-serving-cert\"" Jul 12 18:48:27.193495 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.193471 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8"] Jul 12 18:48:27.194010 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.193987 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" Jul 12 18:48:27.198599 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.198575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30dab25d-0d1a-4719-84e0-00cea6debd35-openshift-service-ca\") pod \"maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8\" (UID: \"30dab25d-0d1a-4719-84e0-00cea6debd35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" Jul 12 18:48:27.198714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.198627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlsnn\" (UniqueName: \"kubernetes.io/projected/30dab25d-0d1a-4719-84e0-00cea6debd35-kube-api-access-hlsnn\") pod \"maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8\" (UID: \"30dab25d-0d1a-4719-84e0-00cea6debd35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" Jul 12 18:48:27.198714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.198689 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30dab25d-0d1a-4719-84e0-00cea6debd35-maas-api-tls\") pod \"maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8\" (UID: \"30dab25d-0d1a-4719-84e0-00cea6debd35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" Jul 12 18:48:27.256050 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.256011 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:27.299687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.299650 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30dab25d-0d1a-4719-84e0-00cea6debd35-openshift-service-ca\") pod \"maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8\" (UID: \"30dab25d-0d1a-4719-84e0-00cea6debd35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" Jul 12 18:48:27.299889 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.299701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlsnn\" (UniqueName: \"kubernetes.io/projected/30dab25d-0d1a-4719-84e0-00cea6debd35-kube-api-access-hlsnn\") pod \"maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8\" (UID: \"30dab25d-0d1a-4719-84e0-00cea6debd35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" Jul 12 18:48:27.299889 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.299729 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30dab25d-0d1a-4719-84e0-00cea6debd35-maas-api-tls\") pod \"maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8\" (UID: \"30dab25d-0d1a-4719-84e0-00cea6debd35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" Jul 12 18:48:27.300031 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:48:27.299893 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-serving-cert: secret "maas-api-e2e-inf-b-89494f-serving-cert" not found Jul 12 18:48:27.300031 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:48:27.299959 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30dab25d-0d1a-4719-84e0-00cea6debd35-maas-api-tls podName:30dab25d-0d1a-4719-84e0-00cea6debd35 nodeName:}" failed. No retries permitted until 2026-07-12 18:48:27.799938429 +0000 UTC m=+1869.567235990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/30dab25d-0d1a-4719-84e0-00cea6debd35-maas-api-tls") pod "maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" (UID: "30dab25d-0d1a-4719-84e0-00cea6debd35") : secret "maas-api-e2e-inf-b-89494f-serving-cert" not found Jul 12 18:48:27.300465 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.300443 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30dab25d-0d1a-4719-84e0-00cea6debd35-openshift-service-ca\") pod \"maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8\" (UID: \"30dab25d-0d1a-4719-84e0-00cea6debd35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" Jul 12 18:48:27.321907 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.321880 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlsnn\" (UniqueName: \"kubernetes.io/projected/30dab25d-0d1a-4719-84e0-00cea6debd35-kube-api-access-hlsnn\") pod \"maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8\" (UID: \"30dab25d-0d1a-4719-84e0-00cea6debd35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" Jul 12 18:48:27.467570 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.467488 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:27.747907 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.747819 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:27.805694 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.805661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30dab25d-0d1a-4719-84e0-00cea6debd35-maas-api-tls\") pod \"maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8\" (UID: \"30dab25d-0d1a-4719-84e0-00cea6debd35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" Jul 12 18:48:27.808212 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.808190 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30dab25d-0d1a-4719-84e0-00cea6debd35-maas-api-tls\") pod \"maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8\" (UID: \"30dab25d-0d1a-4719-84e0-00cea6debd35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" Jul 12 18:48:27.848532 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:27.848500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:28.095895 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:28.095851 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" Jul 12 18:48:28.237934 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:28.237908 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8"] Jul 12 18:48:28.239780 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:48:28.239743 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30dab25d_0d1a_4719_84e0_00cea6debd35.slice/crio-7eebf2eb9ac502311d67a8d1266ba8c39af8092166a60c4959a010f4319d16da WatchSource:0}: Error finding container 7eebf2eb9ac502311d67a8d1266ba8c39af8092166a60c4959a010f4319d16da: Status 404 returned error can't find the container with id 7eebf2eb9ac502311d67a8d1266ba8c39af8092166a60c4959a010f4319d16da Jul 12 18:48:29.226671 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:29.226024 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" event={"ID":"30dab25d-0d1a-4719-84e0-00cea6debd35","Type":"ContainerStarted","Data":"a3fcad543c0e178ae7514da76d99443e76bbbfd427a9ffdd5b19a6fbe6a9d42c"} Jul 12 18:48:29.226671 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:29.226087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" event={"ID":"30dab25d-0d1a-4719-84e0-00cea6debd35","Type":"ContainerStarted","Data":"7eebf2eb9ac502311d67a8d1266ba8c39af8092166a60c4959a010f4319d16da"} Jul 12 18:48:29.226671 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:29.226248 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" Jul 12 18:48:29.243250 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:29.243200 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" podStartSLOduration=1.9455767910000001 podStartE2EDuration="2.24318405s" totalImagesPullingTime="297.607259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:48:29.242399421 +0000 UTC m=+1871.009697005" watchObservedRunningTime="2026-07-12 18:48:29.24318405 +0000 UTC m=+1871.010481632" Jul 12 18:48:31.591678 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:31.591642 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:35.242973 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:35.242942 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" Jul 12 18:48:40.879347 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:40.879264 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:40.980304 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:40.980271 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:41.638638 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:41.638038 2576 patch_prober.go:28] interesting pod/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": read tcp 10.134.0.2:40188->10.134.0.57:15021: read: connection reset by peer" start-of-body= Jul 12 18:48:41.638638 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:41.638115 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" podUID="b04bd126-ba33-4119-858e-3250ed60f0d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": read tcp 10.134.0.2:40188->10.134.0.57:15021: read: connection reset by peer" Jul 12 18:48:42.298157 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:42.298130 2576 generic.go:356] "Generic (PLEG): container finished" podID="b04bd126-ba33-4119-858e-3250ed60f0d0" containerID="28f70be01239131f28dc78eea92f3aacff4bd8104cc8172d03e4f30247cc3223" exitCode=0 Jul 12 18:48:42.298542 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:42.298213 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" event={"ID":"b04bd126-ba33-4119-858e-3250ed60f0d0","Type":"ContainerDied","Data":"28f70be01239131f28dc78eea92f3aacff4bd8104cc8172d03e4f30247cc3223"} Jul 12 18:48:42.298615 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:42.298600 2576 scope.go:117] "RemoveContainer" containerID="28f70be01239131f28dc78eea92f3aacff4bd8104cc8172d03e4f30247cc3223" Jul 12 18:48:42.299619 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:42.299586 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:48:42.299710 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:42.299656 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:48:42.299710 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:42.299687 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:48:43.305458 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:43.305423 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" event={"ID":"b04bd126-ba33-4119-858e-3250ed60f0d0","Type":"ContainerStarted","Data":"dfd16b8adffcdb2fc433758101b1461ffba4c0c8b9cf72ec2409e10dbcfcce70"} Jul 12 18:48:43.730881 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:43.730746 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:43.731515 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:43.731486 2576 patch_prober.go:28] interesting pod/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 12 18:48:43.731702 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:43.731676 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" podUID="b04bd126-ba33-4119-858e-3250ed60f0d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 12 18:48:44.731775 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:44.731737 2576 patch_prober.go:28] interesting pod/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 12 18:48:44.732621 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:44.732584 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" podUID="b04bd126-ba33-4119-858e-3250ed60f0d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 12 18:48:44.989250 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:44.989160 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:45.731460 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:45.731421 2576 patch_prober.go:28] interesting pod/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 12 18:48:45.731738 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:45.731708 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" podUID="b04bd126-ba33-4119-858e-3250ed60f0d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 12 18:48:46.732476 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:46.732023 2576 patch_prober.go:28] interesting pod/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 12 18:48:46.732476 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:46.732091 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" podUID="b04bd126-ba33-4119-858e-3250ed60f0d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 12 18:48:47.735522 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:47.735490 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:47.735927 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:47.735817 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:47.736698 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:47.736678 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:48:49.325659 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:49.325615 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:48:49.425471 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:48:49.425390 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:49:12.798046 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:12.798008 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:49:14.402155 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:14.402116 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:49:29.891388 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:29.891351 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:49:30.833378 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:30.833337 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:49:30.970047 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:30.970006 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:49:31.373181 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:31.373153 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:49:31.473272 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:31.473239 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:49:31.573840 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:31.573802 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:49:31.695468 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:31.695378 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5"] Jul 12 18:49:31.695703 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:31.695670 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" podUID="f3175eb4-b4dc-48d5-831f-b8e09325dccd" containerName="maas-api" containerID="cri-o://129f458f7a24bd204e280e4cfaf088e152425299fea040db1fa1d147badb3b46" gracePeriod=30 Jul 12 18:49:31.720319 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:31.720293 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:49:32.157988 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.157960 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" Jul 12 18:49:32.228443 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.228363 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvzqm\" (UniqueName: \"kubernetes.io/projected/f3175eb4-b4dc-48d5-831f-b8e09325dccd-kube-api-access-qvzqm\") pod \"f3175eb4-b4dc-48d5-831f-b8e09325dccd\" (UID: \"f3175eb4-b4dc-48d5-831f-b8e09325dccd\") " Jul 12 18:49:32.228443 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.228421 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3175eb4-b4dc-48d5-831f-b8e09325dccd-maas-api-tls\") pod \"f3175eb4-b4dc-48d5-831f-b8e09325dccd\" (UID: \"f3175eb4-b4dc-48d5-831f-b8e09325dccd\") " Jul 12 18:49:32.228443 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.228444 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3175eb4-b4dc-48d5-831f-b8e09325dccd-openshift-service-ca\") pod \"f3175eb4-b4dc-48d5-831f-b8e09325dccd\" (UID: \"f3175eb4-b4dc-48d5-831f-b8e09325dccd\") " Jul 12 18:49:32.228907 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.228870 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3175eb4-b4dc-48d5-831f-b8e09325dccd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f3175eb4-b4dc-48d5-831f-b8e09325dccd" (UID: "f3175eb4-b4dc-48d5-831f-b8e09325dccd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:49:32.230564 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.230529 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3175eb4-b4dc-48d5-831f-b8e09325dccd-kube-api-access-qvzqm" (OuterVolumeSpecName: "kube-api-access-qvzqm") pod "f3175eb4-b4dc-48d5-831f-b8e09325dccd" (UID: "f3175eb4-b4dc-48d5-831f-b8e09325dccd"). InnerVolumeSpecName "kube-api-access-qvzqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:49:32.230564 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.230549 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3175eb4-b4dc-48d5-831f-b8e09325dccd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f3175eb4-b4dc-48d5-831f-b8e09325dccd" (UID: "f3175eb4-b4dc-48d5-831f-b8e09325dccd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:49:32.329285 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.329243 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvzqm\" (UniqueName: \"kubernetes.io/projected/f3175eb4-b4dc-48d5-831f-b8e09325dccd-kube-api-access-qvzqm\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:49:32.329285 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.329281 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3175eb4-b4dc-48d5-831f-b8e09325dccd-maas-api-tls\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:49:32.329477 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.329296 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3175eb4-b4dc-48d5-831f-b8e09325dccd-openshift-service-ca\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:49:32.563003 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.562914 2576 generic.go:356] "Generic (PLEG): container finished" podID="f3175eb4-b4dc-48d5-831f-b8e09325dccd" containerID="129f458f7a24bd204e280e4cfaf088e152425299fea040db1fa1d147badb3b46" exitCode=0 Jul 12 18:49:32.563003 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.562956 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" Jul 12 18:49:32.563003 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.562966 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" event={"ID":"f3175eb4-b4dc-48d5-831f-b8e09325dccd","Type":"ContainerDied","Data":"129f458f7a24bd204e280e4cfaf088e152425299fea040db1fa1d147badb3b46"} Jul 12 18:49:32.563261 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.563019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5" event={"ID":"f3175eb4-b4dc-48d5-831f-b8e09325dccd","Type":"ContainerDied","Data":"ab54ba8cc13f4adad77f364360c53937ad27ef9501e20fcd47c75bd91a8863a6"} Jul 12 18:49:32.563261 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.563047 2576 scope.go:117] "RemoveContainer" containerID="129f458f7a24bd204e280e4cfaf088e152425299fea040db1fa1d147badb3b46" Jul 12 18:49:32.572567 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.572544 2576 scope.go:117] "RemoveContainer" containerID="129f458f7a24bd204e280e4cfaf088e152425299fea040db1fa1d147badb3b46" Jul 12 18:49:32.572835 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:49:32.572812 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"129f458f7a24bd204e280e4cfaf088e152425299fea040db1fa1d147badb3b46\": container with ID starting with 129f458f7a24bd204e280e4cfaf088e152425299fea040db1fa1d147badb3b46 not found: ID does not exist" containerID="129f458f7a24bd204e280e4cfaf088e152425299fea040db1fa1d147badb3b46" Jul 12 18:49:32.572906 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.572848 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"129f458f7a24bd204e280e4cfaf088e152425299fea040db1fa1d147badb3b46"} err="failed to get container status \"129f458f7a24bd204e280e4cfaf088e152425299fea040db1fa1d147badb3b46\": rpc error: code = NotFound desc = could not find container \"129f458f7a24bd204e280e4cfaf088e152425299fea040db1fa1d147badb3b46\": container with ID starting with 129f458f7a24bd204e280e4cfaf088e152425299fea040db1fa1d147badb3b46 not found: ID does not exist" Jul 12 18:49:32.584155 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.584129 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5"] Jul 12 18:49:32.587256 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.587234 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-14e82f-7f98588b45-g6jz5"] Jul 12 18:49:32.824484 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:32.824457 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3175eb4-b4dc-48d5-831f-b8e09325dccd" path="/var/lib/kubelet/pods/f3175eb4-b4dc-48d5-831f-b8e09325dccd/volumes" Jul 12 18:49:36.451226 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:49:36.451189 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:13.159645 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:13.159608 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:13.461701 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:13.461602 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:13.834376 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:13.834341 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:14.240271 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.240179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:14.340735 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.340693 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:14.452716 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.452683 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8"] Jul 12 18:50:14.453219 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.453186 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" podUID="30dab25d-0d1a-4719-84e0-00cea6debd35" containerName="maas-api" containerID="cri-o://a3fcad543c0e178ae7514da76d99443e76bbbfd427a9ffdd5b19a6fbe6a9d42c" gracePeriod=30 Jul 12 18:50:14.480593 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.480567 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:14.714140 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.714117 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" Jul 12 18:50:14.793310 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.793266 2576 generic.go:356] "Generic (PLEG): container finished" podID="30dab25d-0d1a-4719-84e0-00cea6debd35" containerID="a3fcad543c0e178ae7514da76d99443e76bbbfd427a9ffdd5b19a6fbe6a9d42c" exitCode=0 Jul 12 18:50:14.793478 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.793314 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" event={"ID":"30dab25d-0d1a-4719-84e0-00cea6debd35","Type":"ContainerDied","Data":"a3fcad543c0e178ae7514da76d99443e76bbbfd427a9ffdd5b19a6fbe6a9d42c"} Jul 12 18:50:14.793478 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.793334 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" Jul 12 18:50:14.793478 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.793386 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8" event={"ID":"30dab25d-0d1a-4719-84e0-00cea6debd35","Type":"ContainerDied","Data":"7eebf2eb9ac502311d67a8d1266ba8c39af8092166a60c4959a010f4319d16da"} Jul 12 18:50:14.793478 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.793410 2576 scope.go:117] "RemoveContainer" containerID="a3fcad543c0e178ae7514da76d99443e76bbbfd427a9ffdd5b19a6fbe6a9d42c" Jul 12 18:50:14.802473 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.802450 2576 scope.go:117] "RemoveContainer" containerID="a3fcad543c0e178ae7514da76d99443e76bbbfd427a9ffdd5b19a6fbe6a9d42c" Jul 12 18:50:14.802762 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:50:14.802742 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3fcad543c0e178ae7514da76d99443e76bbbfd427a9ffdd5b19a6fbe6a9d42c\": container with ID starting with a3fcad543c0e178ae7514da76d99443e76bbbfd427a9ffdd5b19a6fbe6a9d42c not found: ID does not exist" containerID="a3fcad543c0e178ae7514da76d99443e76bbbfd427a9ffdd5b19a6fbe6a9d42c" Jul 12 18:50:14.802840 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.802771 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3fcad543c0e178ae7514da76d99443e76bbbfd427a9ffdd5b19a6fbe6a9d42c"} err="failed to get container status \"a3fcad543c0e178ae7514da76d99443e76bbbfd427a9ffdd5b19a6fbe6a9d42c\": rpc error: code = NotFound desc = could not find container \"a3fcad543c0e178ae7514da76d99443e76bbbfd427a9ffdd5b19a6fbe6a9d42c\": container with ID starting with a3fcad543c0e178ae7514da76d99443e76bbbfd427a9ffdd5b19a6fbe6a9d42c not found: ID does not exist" Jul 12 18:50:14.837750 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.837681 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlsnn\" (UniqueName: \"kubernetes.io/projected/30dab25d-0d1a-4719-84e0-00cea6debd35-kube-api-access-hlsnn\") pod \"30dab25d-0d1a-4719-84e0-00cea6debd35\" (UID: \"30dab25d-0d1a-4719-84e0-00cea6debd35\") " Jul 12 18:50:14.837899 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.837796 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30dab25d-0d1a-4719-84e0-00cea6debd35-openshift-service-ca\") pod \"30dab25d-0d1a-4719-84e0-00cea6debd35\" (UID: \"30dab25d-0d1a-4719-84e0-00cea6debd35\") " Jul 12 18:50:14.837899 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.837825 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30dab25d-0d1a-4719-84e0-00cea6debd35-maas-api-tls\") pod \"30dab25d-0d1a-4719-84e0-00cea6debd35\" (UID: \"30dab25d-0d1a-4719-84e0-00cea6debd35\") " Jul 12 18:50:14.838132 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.838106 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30dab25d-0d1a-4719-84e0-00cea6debd35-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "30dab25d-0d1a-4719-84e0-00cea6debd35" (UID: "30dab25d-0d1a-4719-84e0-00cea6debd35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:50:14.838208 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.838192 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30dab25d-0d1a-4719-84e0-00cea6debd35-openshift-service-ca\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:14.839953 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.839916 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30dab25d-0d1a-4719-84e0-00cea6debd35-kube-api-access-hlsnn" (OuterVolumeSpecName: "kube-api-access-hlsnn") pod "30dab25d-0d1a-4719-84e0-00cea6debd35" (UID: "30dab25d-0d1a-4719-84e0-00cea6debd35"). InnerVolumeSpecName "kube-api-access-hlsnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:50:14.840080 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.840006 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30dab25d-0d1a-4719-84e0-00cea6debd35-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "30dab25d-0d1a-4719-84e0-00cea6debd35" (UID: "30dab25d-0d1a-4719-84e0-00cea6debd35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:50:14.938918 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.938882 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30dab25d-0d1a-4719-84e0-00cea6debd35-maas-api-tls\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:14.938918 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:14.938911 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlsnn\" (UniqueName: \"kubernetes.io/projected/30dab25d-0d1a-4719-84e0-00cea6debd35-kube-api-access-hlsnn\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:15.113674 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:15.113641 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8"] Jul 12 18:50:15.118426 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:15.118400 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89494f-dc75dbc7d-zd2b8"] Jul 12 18:50:16.817681 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:16.817647 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30dab25d-0d1a-4719-84e0-00cea6debd35" path="/var/lib/kubelet/pods/30dab25d-0d1a-4719-84e0-00cea6debd35/volumes" Jul 12 18:50:19.861115 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:19.861074 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:19.917026 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:19.916995 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:20.098155 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:20.098117 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv"] Jul 12 18:50:20.098408 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:20.098354 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" podUID="b04bd126-ba33-4119-858e-3250ed60f0d0" containerName="istio-proxy" containerID="cri-o://dfd16b8adffcdb2fc433758101b1461ffba4c0c8b9cf72ec2409e10dbcfcce70" gracePeriod=30 Jul 12 18:50:20.120094 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:20.120009 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:20.659154 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:20.659125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:20.956605 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:20.956517 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:21.335455 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:21.335420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:23.948635 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:23.948536 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:24.016548 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.016518 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:24.176284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.176248 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr"] Jul 12 18:50:24.176689 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.176673 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3175eb4-b4dc-48d5-831f-b8e09325dccd" containerName="maas-api" Jul 12 18:50:24.176689 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.176687 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3175eb4-b4dc-48d5-831f-b8e09325dccd" containerName="maas-api" Jul 12 18:50:24.176868 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.176694 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30dab25d-0d1a-4719-84e0-00cea6debd35" containerName="maas-api" Jul 12 18:50:24.176868 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.176699 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="30dab25d-0d1a-4719-84e0-00cea6debd35" containerName="maas-api" Jul 12 18:50:24.176868 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.176769 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3175eb4-b4dc-48d5-831f-b8e09325dccd" containerName="maas-api" Jul 12 18:50:24.176868 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.176778 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="30dab25d-0d1a-4719-84e0-00cea6debd35" containerName="maas-api" Jul 12 18:50:24.179948 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.179923 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" Jul 12 18:50:24.182258 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.182237 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-675ba4-serving-cert\"" Jul 12 18:50:24.188315 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.188287 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr"] Jul 12 18:50:24.201410 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.201328 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:24.332700 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.332654 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-maas-api-tls\") pod \"maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr\" (UID: \"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" Jul 12 18:50:24.332913 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.332739 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-openshift-service-ca\") pod \"maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr\" (UID: \"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" Jul 12 18:50:24.332913 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.332886 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dj92\" (UniqueName: \"kubernetes.io/projected/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-kube-api-access-5dj92\") pod \"maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr\" (UID: \"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" Jul 12 18:50:24.434215 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.434174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dj92\" (UniqueName: \"kubernetes.io/projected/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-kube-api-access-5dj92\") pod \"maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr\" (UID: \"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" Jul 12 18:50:24.434429 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.434259 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-maas-api-tls\") pod \"maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr\" (UID: \"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" Jul 12 18:50:24.434429 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.434287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-openshift-service-ca\") pod \"maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr\" (UID: \"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" Jul 12 18:50:24.434429 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:50:24.434396 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-serving-cert: secret "maas-api-e2e-api-a-675ba4-serving-cert" not found Jul 12 18:50:24.434593 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:50:24.434471 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-maas-api-tls podName:50943ed0-e379-4c9f-ae51-0d5d7abcfcc2 nodeName:}" failed. No retries permitted until 2026-07-12 18:50:24.934453826 +0000 UTC m=+1986.701751387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-maas-api-tls") pod "maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" (UID: "50943ed0-e379-4c9f-ae51-0d5d7abcfcc2") : secret "maas-api-e2e-api-a-675ba4-serving-cert" not found Jul 12 18:50:24.434932 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.434914 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-openshift-service-ca\") pod \"maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr\" (UID: \"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" Jul 12 18:50:24.442422 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.442397 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dj92\" (UniqueName: \"kubernetes.io/projected/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-kube-api-access-5dj92\") pod \"maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr\" (UID: \"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" Jul 12 18:50:24.492153 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.492071 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:24.598415 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.598378 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:24.699029 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.698991 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:24.940492 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.940455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-maas-api-tls\") pod \"maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr\" (UID: \"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" Jul 12 18:50:24.943292 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:24.943266 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-maas-api-tls\") pod \"maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr\" (UID: \"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" Jul 12 18:50:25.093122 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.093080 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" Jul 12 18:50:25.238301 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.238270 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr"] Jul 12 18:50:25.240640 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:50:25.240595 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50943ed0_e379_4c9f_ae51_0d5d7abcfcc2.slice/crio-e65705eb1a29f1869b0262a9263a8812f563cb0f334cd3f737760e891f62a9d1 WatchSource:0}: Error finding container e65705eb1a29f1869b0262a9263a8812f563cb0f334cd3f737760e891f62a9d1: Status 404 returned error can't find the container with id e65705eb1a29f1869b0262a9263a8812f563cb0f334cd3f737760e891f62a9d1 Jul 12 18:50:25.367229 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.367202 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:50:25.546292 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.546216 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxrcx\" (UniqueName: \"kubernetes.io/projected/b04bd126-ba33-4119-858e-3250ed60f0d0-kube-api-access-sxrcx\") pod \"b04bd126-ba33-4119-858e-3250ed60f0d0\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " Jul 12 18:50:25.546292 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.546263 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-credential-socket\") pod \"b04bd126-ba33-4119-858e-3250ed60f0d0\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " Jul 12 18:50:25.546481 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.546296 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-data\") pod \"b04bd126-ba33-4119-858e-3250ed60f0d0\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " Jul 12 18:50:25.546481 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.546465 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04bd126-ba33-4119-858e-3250ed60f0d0-istiod-ca-cert\") pod \"b04bd126-ba33-4119-858e-3250ed60f0d0\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " Jul 12 18:50:25.546571 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.546516 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-podinfo\") pod \"b04bd126-ba33-4119-858e-3250ed60f0d0\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " Jul 12 18:50:25.546571 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.546547 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-workload-certs\") pod \"b04bd126-ba33-4119-858e-3250ed60f0d0\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " Jul 12 18:50:25.546678 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.546581 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b04bd126-ba33-4119-858e-3250ed60f0d0" (UID: "b04bd126-ba33-4119-858e-3250ed60f0d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:50:25.546678 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.546605 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-envoy\") pod \"b04bd126-ba33-4119-858e-3250ed60f0d0\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " Jul 12 18:50:25.546678 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.546637 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-token\") pod \"b04bd126-ba33-4119-858e-3250ed60f0d0\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " Jul 12 18:50:25.546874 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.546681 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-workload-socket\") pod \"b04bd126-ba33-4119-858e-3250ed60f0d0\" (UID: \"b04bd126-ba33-4119-858e-3250ed60f0d0\") " Jul 12 18:50:25.546874 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.546801 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b04bd126-ba33-4119-858e-3250ed60f0d0" (UID: "b04bd126-ba33-4119-858e-3250ed60f0d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:50:25.546977 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.546912 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04bd126-ba33-4119-858e-3250ed60f0d0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b04bd126-ba33-4119-858e-3250ed60f0d0" (UID: "b04bd126-ba33-4119-858e-3250ed60f0d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:50:25.547070 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.547035 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b04bd126-ba33-4119-858e-3250ed60f0d0" (UID: "b04bd126-ba33-4119-858e-3250ed60f0d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:50:25.547224 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.547194 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-credential-socket\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:25.547359 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.547232 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04bd126-ba33-4119-858e-3250ed60f0d0-istiod-ca-cert\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:25.547359 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.547247 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-workload-certs\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:25.547359 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.547259 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-workload-socket\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:25.547359 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.547198 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-data" (OuterVolumeSpecName: "istio-data") pod "b04bd126-ba33-4119-858e-3250ed60f0d0" (UID: "b04bd126-ba33-4119-858e-3250ed60f0d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:50:25.548628 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.548602 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04bd126-ba33-4119-858e-3250ed60f0d0-kube-api-access-sxrcx" (OuterVolumeSpecName: "kube-api-access-sxrcx") pod "b04bd126-ba33-4119-858e-3250ed60f0d0" (UID: "b04bd126-ba33-4119-858e-3250ed60f0d0"). InnerVolumeSpecName "kube-api-access-sxrcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:50:25.548732 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.548646 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b04bd126-ba33-4119-858e-3250ed60f0d0" (UID: "b04bd126-ba33-4119-858e-3250ed60f0d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 18:50:25.549220 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.549198 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "b04bd126-ba33-4119-858e-3250ed60f0d0" (UID: "b04bd126-ba33-4119-858e-3250ed60f0d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:50:25.549304 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.549249 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b04bd126-ba33-4119-858e-3250ed60f0d0" (UID: "b04bd126-ba33-4119-858e-3250ed60f0d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:50:25.648100 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.648067 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-data\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:25.648100 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.648096 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-podinfo\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:25.648100 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.648105 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-envoy\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:25.648392 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.648112 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04bd126-ba33-4119-858e-3250ed60f0d0-istio-token\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:25.648392 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.648120 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxrcx\" (UniqueName: \"kubernetes.io/projected/b04bd126-ba33-4119-858e-3250ed60f0d0-kube-api-access-sxrcx\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:25.855776 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.855742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" event={"ID":"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2","Type":"ContainerStarted","Data":"7b6efdc58c8531e62cb3d81b30b56e7785f683b02b95ff439a2223d73455e70b"} Jul 12 18:50:25.855984 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.855824 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" event={"ID":"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2","Type":"ContainerStarted","Data":"e65705eb1a29f1869b0262a9263a8812f563cb0f334cd3f737760e891f62a9d1"} Jul 12 18:50:25.855984 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.855856 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" Jul 12 18:50:25.857319 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.857291 2576 generic.go:356] "Generic (PLEG): container finished" podID="b04bd126-ba33-4119-858e-3250ed60f0d0" containerID="dfd16b8adffcdb2fc433758101b1461ffba4c0c8b9cf72ec2409e10dbcfcce70" exitCode=0 Jul 12 18:50:25.857440 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.857337 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" Jul 12 18:50:25.857440 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.857369 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" event={"ID":"b04bd126-ba33-4119-858e-3250ed60f0d0","Type":"ContainerDied","Data":"dfd16b8adffcdb2fc433758101b1461ffba4c0c8b9cf72ec2409e10dbcfcce70"} Jul 12 18:50:25.857440 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.857419 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv" event={"ID":"b04bd126-ba33-4119-858e-3250ed60f0d0","Type":"ContainerDied","Data":"3bd69b9ae506639a0a696859ca9408a6a5b50e6d22330f0405d159ebeb524773"} Jul 12 18:50:25.857555 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.857443 2576 scope.go:117] "RemoveContainer" containerID="dfd16b8adffcdb2fc433758101b1461ffba4c0c8b9cf72ec2409e10dbcfcce70" Jul 12 18:50:25.867177 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.867159 2576 scope.go:117] "RemoveContainer" containerID="28f70be01239131f28dc78eea92f3aacff4bd8104cc8172d03e4f30247cc3223" Jul 12 18:50:25.873714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.873625 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" podStartSLOduration=1.365745373 podStartE2EDuration="1.873611727s" totalImagesPullingTime="507.866354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:50:25.870969051 +0000 UTC m=+1987.638266633" watchObservedRunningTime="2026-07-12 18:50:25.873611727 +0000 UTC m=+1987.640909308" Jul 12 18:50:25.876017 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.875999 2576 scope.go:117] "RemoveContainer" containerID="dfd16b8adffcdb2fc433758101b1461ffba4c0c8b9cf72ec2409e10dbcfcce70" Jul 12 18:50:25.876258 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:50:25.876238 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd16b8adffcdb2fc433758101b1461ffba4c0c8b9cf72ec2409e10dbcfcce70\": container with ID starting with dfd16b8adffcdb2fc433758101b1461ffba4c0c8b9cf72ec2409e10dbcfcce70 not found: ID does not exist" containerID="dfd16b8adffcdb2fc433758101b1461ffba4c0c8b9cf72ec2409e10dbcfcce70" Jul 12 18:50:25.876297 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.876266 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd16b8adffcdb2fc433758101b1461ffba4c0c8b9cf72ec2409e10dbcfcce70"} err="failed to get container status \"dfd16b8adffcdb2fc433758101b1461ffba4c0c8b9cf72ec2409e10dbcfcce70\": rpc error: code = NotFound desc = could not find container \"dfd16b8adffcdb2fc433758101b1461ffba4c0c8b9cf72ec2409e10dbcfcce70\": container with ID starting with dfd16b8adffcdb2fc433758101b1461ffba4c0c8b9cf72ec2409e10dbcfcce70 not found: ID does not exist" Jul 12 18:50:25.876297 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.876294 2576 scope.go:117] "RemoveContainer" containerID="28f70be01239131f28dc78eea92f3aacff4bd8104cc8172d03e4f30247cc3223" Jul 12 18:50:25.876534 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:50:25.876517 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f70be01239131f28dc78eea92f3aacff4bd8104cc8172d03e4f30247cc3223\": container with ID starting with 28f70be01239131f28dc78eea92f3aacff4bd8104cc8172d03e4f30247cc3223 not found: ID does not exist" containerID="28f70be01239131f28dc78eea92f3aacff4bd8104cc8172d03e4f30247cc3223" Jul 12 18:50:25.876572 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.876541 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f70be01239131f28dc78eea92f3aacff4bd8104cc8172d03e4f30247cc3223"} err="failed to get container status \"28f70be01239131f28dc78eea92f3aacff4bd8104cc8172d03e4f30247cc3223\": rpc error: code = NotFound desc = could not find container \"28f70be01239131f28dc78eea92f3aacff4bd8104cc8172d03e4f30247cc3223\": container with ID starting with 28f70be01239131f28dc78eea92f3aacff4bd8104cc8172d03e4f30247cc3223 not found: ID does not exist" Jul 12 18:50:25.884250 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.884230 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv"] Jul 12 18:50:25.887509 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:25.887489 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-89494f-openshift-default-6bc8c679f5-4n7cv"] Jul 12 18:50:26.818795 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:26.818763 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b04bd126-ba33-4119-858e-3250ed60f0d0" path="/var/lib/kubelet/pods/b04bd126-ba33-4119-858e-3250ed60f0d0/volumes" Jul 12 18:50:28.445022 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:28.444994 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:28.876467 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:28.876428 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:29.146110 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.146027 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps"] Jul 12 18:50:29.146719 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.146698 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b04bd126-ba33-4119-858e-3250ed60f0d0" containerName="istio-proxy" Jul 12 18:50:29.146763 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.146725 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b04bd126-ba33-4119-858e-3250ed60f0d0" containerName="istio-proxy" Jul 12 18:50:29.146822 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.146761 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b04bd126-ba33-4119-858e-3250ed60f0d0" containerName="istio-proxy" Jul 12 18:50:29.146822 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.146770 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b04bd126-ba33-4119-858e-3250ed60f0d0" containerName="istio-proxy" Jul 12 18:50:29.146908 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.146894 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b04bd126-ba33-4119-858e-3250ed60f0d0" containerName="istio-proxy" Jul 12 18:50:29.146940 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.146912 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b04bd126-ba33-4119-858e-3250ed60f0d0" containerName="istio-proxy" Jul 12 18:50:29.151991 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.151974 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.154396 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.154377 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-443d18-openshift-default-dockercfg-9jsvl\"" Jul 12 18:50:29.159882 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.159584 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps"] Jul 12 18:50:29.170272 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.170223 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:29.283366 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.283328 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-token\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.283550 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.283451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istiod-ca-cert\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.283550 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.283481 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-workload-certs\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.283655 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.283566 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-credential-socket\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.283655 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.283625 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-data\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.283655 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.283644 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-workload-socket\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.283830 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.283684 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-podinfo\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.283830 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.283746 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-envoy\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.283830 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.283771 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjwhn\" (UniqueName: \"kubernetes.io/projected/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-kube-api-access-fjwhn\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.385068 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.385033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-credential-socket\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.385242 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.385094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-data\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.385242 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.385114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-workload-socket\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.385242 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.385133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-podinfo\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.385242 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.385152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-envoy\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.385242 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.385167 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjwhn\" (UniqueName: \"kubernetes.io/projected/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-kube-api-access-fjwhn\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.385242 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.385200 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-token\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.385899 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.385260 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istiod-ca-cert\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.385899 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.385286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-workload-certs\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.385899 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.385500 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-credential-socket\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.385899 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.385545 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-workload-socket\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.385899 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.385848 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-data\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.386090 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.385930 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-workload-certs\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.386090 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.386065 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istiod-ca-cert\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.387508 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.387487 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-envoy\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.387688 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.387672 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-podinfo\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.396531 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.396464 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-token\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.396891 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.396870 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjwhn\" (UniqueName: \"kubernetes.io/projected/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-kube-api-access-fjwhn\") pod \"e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.442155 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.442119 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:29.463936 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.463908 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:29.604917 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.604881 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps"] Jul 12 18:50:29.607829 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:50:29.607793 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aac6669_afc7_4ca4_a3e5_afe457f4b9ed.slice/crio-00125224b3e8d5de509566c9cd93d0c2f6f77450ef7e40b62677379979c8898c WatchSource:0}: Error finding container 00125224b3e8d5de509566c9cd93d0c2f6f77450ef7e40b62677379979c8898c: Status 404 returned error can't find the container with id 00125224b3e8d5de509566c9cd93d0c2f6f77450ef7e40b62677379979c8898c Jul 12 18:50:29.610063 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.610032 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:50:29.610140 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.610097 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:50:29.610140 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.610125 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:50:29.887916 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.887878 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" event={"ID":"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed","Type":"ContainerStarted","Data":"adbaf4b2c0c55a58e2a8fa61750fa0afe7e86fb5841f83ec8e62ecf55df8dd0e"} Jul 12 18:50:29.888068 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.887954 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" event={"ID":"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed","Type":"ContainerStarted","Data":"00125224b3e8d5de509566c9cd93d0c2f6f77450ef7e40b62677379979c8898c"} Jul 12 18:50:29.925223 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:29.925133 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" podStartSLOduration=0.925117864 podStartE2EDuration="925.117864ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:50:29.922450834 +0000 UTC m=+1991.689748417" watchObservedRunningTime="2026-07-12 18:50:29.925117864 +0000 UTC m=+1991.692415446" Jul 12 18:50:30.464945 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:30.464910 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:30.469619 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:30.469595 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:30.892915 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:30.892876 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:30.893890 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:30.893870 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:31.803348 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:31.803298 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:31.868115 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:31.868086 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" Jul 12 18:50:31.887650 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:31.887618 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:31.999775 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:31.999743 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9"] Jul 12 18:50:32.004434 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.004407 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" Jul 12 18:50:32.006880 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.006862 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-443d18-serving-cert\"" Jul 12 18:50:32.007450 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.007430 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080fe60f-03e2-49f0-8309-1869d0c90154-openshift-service-ca\") pod \"maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9\" (UID: \"080fe60f-03e2-49f0-8309-1869d0c90154\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" Jul 12 18:50:32.007532 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.007480 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080fe60f-03e2-49f0-8309-1869d0c90154-maas-api-tls\") pod \"maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9\" (UID: \"080fe60f-03e2-49f0-8309-1869d0c90154\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" Jul 12 18:50:32.007580 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.007536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx8lf\" (UniqueName: \"kubernetes.io/projected/080fe60f-03e2-49f0-8309-1869d0c90154-kube-api-access-dx8lf\") pod \"maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9\" (UID: \"080fe60f-03e2-49f0-8309-1869d0c90154\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" Jul 12 18:50:32.012964 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.012924 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9"] Jul 12 18:50:32.032398 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.032353 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:32.108492 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.108462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080fe60f-03e2-49f0-8309-1869d0c90154-openshift-service-ca\") pod \"maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9\" (UID: \"080fe60f-03e2-49f0-8309-1869d0c90154\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" Jul 12 18:50:32.108661 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.108507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080fe60f-03e2-49f0-8309-1869d0c90154-maas-api-tls\") pod \"maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9\" (UID: \"080fe60f-03e2-49f0-8309-1869d0c90154\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" Jul 12 18:50:32.108661 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.108532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx8lf\" (UniqueName: \"kubernetes.io/projected/080fe60f-03e2-49f0-8309-1869d0c90154-kube-api-access-dx8lf\") pod \"maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9\" (UID: \"080fe60f-03e2-49f0-8309-1869d0c90154\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" Jul 12 18:50:32.108738 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:50:32.108663 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-serving-cert: secret "maas-api-e2e-api-b-443d18-serving-cert" not found Jul 12 18:50:32.108778 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:50:32.108735 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/080fe60f-03e2-49f0-8309-1869d0c90154-maas-api-tls podName:080fe60f-03e2-49f0-8309-1869d0c90154 nodeName:}" failed. No retries permitted until 2026-07-12 18:50:32.608713479 +0000 UTC m=+1994.376011047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/080fe60f-03e2-49f0-8309-1869d0c90154-maas-api-tls") pod "maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" (UID: "080fe60f-03e2-49f0-8309-1869d0c90154") : secret "maas-api-e2e-api-b-443d18-serving-cert" not found Jul 12 18:50:32.109190 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.109173 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080fe60f-03e2-49f0-8309-1869d0c90154-openshift-service-ca\") pod \"maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9\" (UID: \"080fe60f-03e2-49f0-8309-1869d0c90154\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" Jul 12 18:50:32.121329 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.121304 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx8lf\" (UniqueName: \"kubernetes.io/projected/080fe60f-03e2-49f0-8309-1869d0c90154-kube-api-access-dx8lf\") pod \"maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9\" (UID: \"080fe60f-03e2-49f0-8309-1869d0c90154\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" Jul 12 18:50:32.187146 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.187111 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:32.455028 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.454932 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:32.560327 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.560287 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:32.612110 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.612078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080fe60f-03e2-49f0-8309-1869d0c90154-maas-api-tls\") pod \"maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9\" (UID: \"080fe60f-03e2-49f0-8309-1869d0c90154\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" Jul 12 18:50:32.614972 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.614932 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080fe60f-03e2-49f0-8309-1869d0c90154-maas-api-tls\") pod \"maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9\" (UID: \"080fe60f-03e2-49f0-8309-1869d0c90154\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" Jul 12 18:50:32.617814 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.617759 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" Jul 12 18:50:32.787216 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.787181 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9"] Jul 12 18:50:32.789633 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:50:32.789590 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080fe60f_03e2_49f0_8309_1869d0c90154.slice/crio-729999826bf09fa03343de966f31a503cad3b139942de6dd532bf875694ca0c4 WatchSource:0}: Error finding container 729999826bf09fa03343de966f31a503cad3b139942de6dd532bf875694ca0c4: Status 404 returned error can't find the container with id 729999826bf09fa03343de966f31a503cad3b139942de6dd532bf875694ca0c4 Jul 12 18:50:32.905568 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:32.905536 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" event={"ID":"080fe60f-03e2-49f0-8309-1869d0c90154","Type":"ContainerStarted","Data":"729999826bf09fa03343de966f31a503cad3b139942de6dd532bf875694ca0c4"} Jul 12 18:50:33.912361 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:33.912324 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" event={"ID":"080fe60f-03e2-49f0-8309-1869d0c90154","Type":"ContainerStarted","Data":"c1ab15f25f00033c12d14aabfd251d19a004836f291073457a80b0a8df685eff"} Jul 12 18:50:33.912769 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:33.912389 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" Jul 12 18:50:33.930839 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:33.930776 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" podStartSLOduration=2.558813758 podStartE2EDuration="2.930759278s" totalImagesPullingTime="371.94552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:50:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:50:33.929982686 +0000 UTC m=+1995.697280281" watchObservedRunningTime="2026-07-12 18:50:33.930759278 +0000 UTC m=+1995.698056861" Jul 12 18:50:36.282899 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:36.282807 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:39.923047 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:39.923021 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" Jul 12 18:50:44.220026 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:44.219989 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:44.600439 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:44.600408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:44.811878 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:44.811840 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:44.913359 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:44.913326 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:45.041658 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.041624 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr"] Jul 12 18:50:45.041901 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.041876 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" podUID="50943ed0-e379-4c9f-ae51-0d5d7abcfcc2" containerName="maas-api" containerID="cri-o://7b6efdc58c8531e62cb3d81b30b56e7785f683b02b95ff439a2223d73455e70b" gracePeriod=30 Jul 12 18:50:45.069866 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.069841 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:45.305237 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.305209 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" Jul 12 18:50:45.433488 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.433398 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-openshift-service-ca\") pod \"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2\" (UID: \"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2\") " Jul 12 18:50:45.433488 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.433445 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dj92\" (UniqueName: \"kubernetes.io/projected/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-kube-api-access-5dj92\") pod \"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2\" (UID: \"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2\") " Jul 12 18:50:45.433703 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.433497 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-maas-api-tls\") pod \"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2\" (UID: \"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2\") " Jul 12 18:50:45.433977 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.433777 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "50943ed0-e379-4c9f-ae51-0d5d7abcfcc2" (UID: "50943ed0-e379-4c9f-ae51-0d5d7abcfcc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:50:45.436092 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.436065 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-kube-api-access-5dj92" (OuterVolumeSpecName: "kube-api-access-5dj92") pod "50943ed0-e379-4c9f-ae51-0d5d7abcfcc2" (UID: "50943ed0-e379-4c9f-ae51-0d5d7abcfcc2"). InnerVolumeSpecName "kube-api-access-5dj92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:50:45.436222 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.436181 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "50943ed0-e379-4c9f-ae51-0d5d7abcfcc2" (UID: "50943ed0-e379-4c9f-ae51-0d5d7abcfcc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:50:45.534625 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.534582 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-maas-api-tls\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:45.534625 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.534621 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-openshift-service-ca\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:45.534903 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.534635 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dj92\" (UniqueName: \"kubernetes.io/projected/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2-kube-api-access-5dj92\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:45.979395 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.979360 2576 generic.go:356] "Generic (PLEG): container finished" podID="50943ed0-e379-4c9f-ae51-0d5d7abcfcc2" containerID="7b6efdc58c8531e62cb3d81b30b56e7785f683b02b95ff439a2223d73455e70b" exitCode=0 Jul 12 18:50:45.979605 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.979417 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" Jul 12 18:50:45.979605 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.979435 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" event={"ID":"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2","Type":"ContainerDied","Data":"7b6efdc58c8531e62cb3d81b30b56e7785f683b02b95ff439a2223d73455e70b"} Jul 12 18:50:45.979605 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.979489 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr" event={"ID":"50943ed0-e379-4c9f-ae51-0d5d7abcfcc2","Type":"ContainerDied","Data":"e65705eb1a29f1869b0262a9263a8812f563cb0f334cd3f737760e891f62a9d1"} Jul 12 18:50:45.979605 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.979510 2576 scope.go:117] "RemoveContainer" containerID="7b6efdc58c8531e62cb3d81b30b56e7785f683b02b95ff439a2223d73455e70b" Jul 12 18:50:45.989573 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.989555 2576 scope.go:117] "RemoveContainer" containerID="7b6efdc58c8531e62cb3d81b30b56e7785f683b02b95ff439a2223d73455e70b" Jul 12 18:50:45.989890 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:50:45.989860 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6efdc58c8531e62cb3d81b30b56e7785f683b02b95ff439a2223d73455e70b\": container with ID starting with 7b6efdc58c8531e62cb3d81b30b56e7785f683b02b95ff439a2223d73455e70b not found: ID does not exist" containerID="7b6efdc58c8531e62cb3d81b30b56e7785f683b02b95ff439a2223d73455e70b" Jul 12 18:50:45.989987 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:45.989896 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6efdc58c8531e62cb3d81b30b56e7785f683b02b95ff439a2223d73455e70b"} err="failed to get container status \"7b6efdc58c8531e62cb3d81b30b56e7785f683b02b95ff439a2223d73455e70b\": rpc error: code = NotFound desc = could not find container \"7b6efdc58c8531e62cb3d81b30b56e7785f683b02b95ff439a2223d73455e70b\": container with ID starting with 7b6efdc58c8531e62cb3d81b30b56e7785f683b02b95ff439a2223d73455e70b not found: ID does not exist" Jul 12 18:50:46.002013 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:46.001980 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr"] Jul 12 18:50:46.008588 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:46.008560 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-675ba4-5575fdcd5-rt7zr"] Jul 12 18:50:46.818169 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:46.818136 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50943ed0-e379-4c9f-ae51-0d5d7abcfcc2" path="/var/lib/kubelet/pods/50943ed0-e379-4c9f-ae51-0d5d7abcfcc2/volumes" Jul 12 18:50:46.933948 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:46.933905 2576 patch_prober.go:28] interesting pod/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 12 18:50:46.934110 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:46.933981 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" podUID="1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 12 18:50:47.993770 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:47.993683 2576 generic.go:356] "Generic (PLEG): container finished" podID="1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" containerID="adbaf4b2c0c55a58e2a8fa61750fa0afe7e86fb5841f83ec8e62ecf55df8dd0e" exitCode=0 Jul 12 18:50:47.993770 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:47.993750 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" event={"ID":"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed","Type":"ContainerDied","Data":"adbaf4b2c0c55a58e2a8fa61750fa0afe7e86fb5841f83ec8e62ecf55df8dd0e"} Jul 12 18:50:47.994267 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:47.994195 2576 scope.go:117] "RemoveContainer" containerID="adbaf4b2c0c55a58e2a8fa61750fa0afe7e86fb5841f83ec8e62ecf55df8dd0e" Jul 12 18:50:47.995196 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:47.995170 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:50:47.995260 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:47.995237 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:50:47.995303 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:47.995282 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:50:49.001004 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:49.000971 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" event={"ID":"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed","Type":"ContainerStarted","Data":"9890f9f03b11dc6fce712b43d38546629e63b82ef9ed62a80b981b472749715d"} Jul 12 18:50:49.465000 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:49.464956 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:49.465189 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:49.465167 2576 patch_prober.go:28] interesting pod/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 12 18:50:49.465244 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:49.465224 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" podUID="1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 12 18:50:50.111572 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:50.111536 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:50.192191 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:50.192151 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:50.366431 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:50.366246 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:50.464628 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:50.464590 2576 patch_prober.go:28] interesting pod/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 12 18:50:50.464834 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:50.464655 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" podUID="1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 12 18:50:50.876006 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:50.875961 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:51.234383 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:51.234292 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:51.411641 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:51.411607 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:51.464776 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:51.464729 2576 patch_prober.go:28] interesting pod/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 12 18:50:51.464952 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:51.464813 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" podUID="1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 12 18:50:51.512852 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:51.512750 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:51.653809 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:51.651701 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9"] Jul 12 18:50:51.657812 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:51.657462 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" podUID="080fe60f-03e2-49f0-8309-1869d0c90154" containerName="maas-api" containerID="cri-o://c1ab15f25f00033c12d14aabfd251d19a004836f291073457a80b0a8df685eff" gracePeriod=30 Jul 12 18:50:51.672324 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:51.671891 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:51.962584 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:51.962551 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" Jul 12 18:50:51.998073 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:51.998040 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080fe60f-03e2-49f0-8309-1869d0c90154-openshift-service-ca\") pod \"080fe60f-03e2-49f0-8309-1869d0c90154\" (UID: \"080fe60f-03e2-49f0-8309-1869d0c90154\") " Jul 12 18:50:51.998246 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:51.998105 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080fe60f-03e2-49f0-8309-1869d0c90154-maas-api-tls\") pod \"080fe60f-03e2-49f0-8309-1869d0c90154\" (UID: \"080fe60f-03e2-49f0-8309-1869d0c90154\") " Jul 12 18:50:51.998318 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:51.998264 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx8lf\" (UniqueName: \"kubernetes.io/projected/080fe60f-03e2-49f0-8309-1869d0c90154-kube-api-access-dx8lf\") pod \"080fe60f-03e2-49f0-8309-1869d0c90154\" (UID: \"080fe60f-03e2-49f0-8309-1869d0c90154\") " Jul 12 18:50:51.998487 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:51.998464 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080fe60f-03e2-49f0-8309-1869d0c90154-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "080fe60f-03e2-49f0-8309-1869d0c90154" (UID: "080fe60f-03e2-49f0-8309-1869d0c90154"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:50:51.998569 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:51.998558 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080fe60f-03e2-49f0-8309-1869d0c90154-openshift-service-ca\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:52.000256 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:52.000232 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080fe60f-03e2-49f0-8309-1869d0c90154-kube-api-access-dx8lf" (OuterVolumeSpecName: "kube-api-access-dx8lf") pod "080fe60f-03e2-49f0-8309-1869d0c90154" (UID: "080fe60f-03e2-49f0-8309-1869d0c90154"). InnerVolumeSpecName "kube-api-access-dx8lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:50:52.000433 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:52.000410 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080fe60f-03e2-49f0-8309-1869d0c90154-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "080fe60f-03e2-49f0-8309-1869d0c90154" (UID: "080fe60f-03e2-49f0-8309-1869d0c90154"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:50:52.018111 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:52.018085 2576 generic.go:356] "Generic (PLEG): container finished" podID="080fe60f-03e2-49f0-8309-1869d0c90154" containerID="c1ab15f25f00033c12d14aabfd251d19a004836f291073457a80b0a8df685eff" exitCode=0 Jul 12 18:50:52.018215 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:52.018127 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" Jul 12 18:50:52.018215 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:52.018164 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" event={"ID":"080fe60f-03e2-49f0-8309-1869d0c90154","Type":"ContainerDied","Data":"c1ab15f25f00033c12d14aabfd251d19a004836f291073457a80b0a8df685eff"} Jul 12 18:50:52.018288 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:52.018238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9" event={"ID":"080fe60f-03e2-49f0-8309-1869d0c90154","Type":"ContainerDied","Data":"729999826bf09fa03343de966f31a503cad3b139942de6dd532bf875694ca0c4"} Jul 12 18:50:52.018288 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:52.018261 2576 scope.go:117] "RemoveContainer" containerID="c1ab15f25f00033c12d14aabfd251d19a004836f291073457a80b0a8df685eff" Jul 12 18:50:52.055414 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:52.055113 2576 scope.go:117] "RemoveContainer" containerID="c1ab15f25f00033c12d14aabfd251d19a004836f291073457a80b0a8df685eff" Jul 12 18:50:52.056215 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:50:52.056118 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ab15f25f00033c12d14aabfd251d19a004836f291073457a80b0a8df685eff\": container with ID starting with c1ab15f25f00033c12d14aabfd251d19a004836f291073457a80b0a8df685eff not found: ID does not exist" containerID="c1ab15f25f00033c12d14aabfd251d19a004836f291073457a80b0a8df685eff" Jul 12 18:50:52.056215 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:52.056159 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ab15f25f00033c12d14aabfd251d19a004836f291073457a80b0a8df685eff"} err="failed to get container status \"c1ab15f25f00033c12d14aabfd251d19a004836f291073457a80b0a8df685eff\": rpc error: code = NotFound desc = could not find container \"c1ab15f25f00033c12d14aabfd251d19a004836f291073457a80b0a8df685eff\": container with ID starting with c1ab15f25f00033c12d14aabfd251d19a004836f291073457a80b0a8df685eff not found: ID does not exist" Jul 12 18:50:52.070022 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:52.069996 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9"] Jul 12 18:50:52.075368 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:52.075340 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-443d18-dbfdd4dbf-455k9"] Jul 12 18:50:52.100051 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:52.100027 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx8lf\" (UniqueName: \"kubernetes.io/projected/080fe60f-03e2-49f0-8309-1869d0c90154-kube-api-access-dx8lf\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:52.100111 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:52.100054 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080fe60f-03e2-49f0-8309-1869d0c90154-maas-api-tls\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:50:52.465075 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:52.465026 2576 patch_prober.go:28] interesting pod/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 12 18:50:52.465454 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:52.465091 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" podUID="1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 12 18:50:52.818486 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:52.818401 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="080fe60f-03e2-49f0-8309-1869d0c90154" path="/var/lib/kubelet/pods/080fe60f-03e2-49f0-8309-1869d0c90154/volumes" Jul 12 18:50:53.468533 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:53.468502 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:53.468946 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:53.468890 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:53.469531 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:53.469513 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:50:56.922515 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:56.922465 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:56.982407 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:56.982377 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:57.159541 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:57.159507 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps"] Jul 12 18:50:57.159709 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:57.159687 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" podUID="1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" containerName="istio-proxy" containerID="cri-o://9890f9f03b11dc6fce712b43d38546629e63b82ef9ed62a80b981b472749715d" gracePeriod=30 Jul 12 18:50:57.173389 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:57.173319 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:57.721542 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:57.721502 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:58.011859 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:58.011748 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:50:58.267329 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:50:58.267244 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:00.043848 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:00.043764 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:00.145630 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:00.145586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:01.202055 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.202018 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:01.263490 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.263456 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh"] Jul 12 18:51:01.264115 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.264087 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50943ed0-e379-4c9f-ae51-0d5d7abcfcc2" containerName="maas-api" Jul 12 18:51:01.264115 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.264114 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50943ed0-e379-4c9f-ae51-0d5d7abcfcc2" containerName="maas-api" Jul 12 18:51:01.264383 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.264142 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="080fe60f-03e2-49f0-8309-1869d0c90154" containerName="maas-api" Jul 12 18:51:01.264383 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.264158 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="080fe60f-03e2-49f0-8309-1869d0c90154" containerName="maas-api" Jul 12 18:51:01.264383 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.264283 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="080fe60f-03e2-49f0-8309-1869d0c90154" containerName="maas-api" Jul 12 18:51:01.264383 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.264295 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="50943ed0-e379-4c9f-ae51-0d5d7abcfcc2" containerName="maas-api" Jul 12 18:51:01.267690 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.267669 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" Jul 12 18:51:01.269978 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.269958 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-1a89be-serving-cert\"" Jul 12 18:51:01.276216 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.276030 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh"] Jul 12 18:51:01.276943 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.276905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:01.348078 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.348041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:01.393360 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.393318 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/391e7e64-1f08-4696-90d6-71af43c4fe97-maas-api-tls\") pod \"maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh\" (UID: \"391e7e64-1f08-4696-90d6-71af43c4fe97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" Jul 12 18:51:01.393508 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.393371 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/391e7e64-1f08-4696-90d6-71af43c4fe97-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh\" (UID: \"391e7e64-1f08-4696-90d6-71af43c4fe97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" Jul 12 18:51:01.393508 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.393483 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdsf9\" (UniqueName: \"kubernetes.io/projected/391e7e64-1f08-4696-90d6-71af43c4fe97-kube-api-access-sdsf9\") pod \"maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh\" (UID: \"391e7e64-1f08-4696-90d6-71af43c4fe97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" Jul 12 18:51:01.468404 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.468311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:01.494079 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.494045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/391e7e64-1f08-4696-90d6-71af43c4fe97-maas-api-tls\") pod \"maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh\" (UID: \"391e7e64-1f08-4696-90d6-71af43c4fe97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" Jul 12 18:51:01.494228 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.494097 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/391e7e64-1f08-4696-90d6-71af43c4fe97-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh\" (UID: \"391e7e64-1f08-4696-90d6-71af43c4fe97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" Jul 12 18:51:01.494228 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.494153 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdsf9\" (UniqueName: \"kubernetes.io/projected/391e7e64-1f08-4696-90d6-71af43c4fe97-kube-api-access-sdsf9\") pod \"maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh\" (UID: \"391e7e64-1f08-4696-90d6-71af43c4fe97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" Jul 12 18:51:01.494770 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.494746 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/391e7e64-1f08-4696-90d6-71af43c4fe97-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh\" (UID: \"391e7e64-1f08-4696-90d6-71af43c4fe97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" Jul 12 18:51:01.496515 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.496490 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/391e7e64-1f08-4696-90d6-71af43c4fe97-maas-api-tls\") pod \"maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh\" (UID: \"391e7e64-1f08-4696-90d6-71af43c4fe97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" Jul 12 18:51:01.502212 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.502191 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdsf9\" (UniqueName: \"kubernetes.io/projected/391e7e64-1f08-4696-90d6-71af43c4fe97-kube-api-access-sdsf9\") pod \"maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh\" (UID: \"391e7e64-1f08-4696-90d6-71af43c4fe97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" Jul 12 18:51:01.581254 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.581223 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" Jul 12 18:51:01.730246 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.730118 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:01.734013 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.733981 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh"] Jul 12 18:51:01.736130 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:51:01.736091 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391e7e64_1f08_4696_90d6_71af43c4fe97.slice/crio-1c0ecedca46386125d3c09628fbb1b94b36278a3a8fb7dd645f7cdabb0114bd4 WatchSource:0}: Error finding container 1c0ecedca46386125d3c09628fbb1b94b36278a3a8fb7dd645f7cdabb0114bd4: Status 404 returned error can't find the container with id 1c0ecedca46386125d3c09628fbb1b94b36278a3a8fb7dd645f7cdabb0114bd4 Jul 12 18:51:01.825828 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:01.825774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:02.073082 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.073041 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" event={"ID":"391e7e64-1f08-4696-90d6-71af43c4fe97","Type":"ContainerStarted","Data":"1c0ecedca46386125d3c09628fbb1b94b36278a3a8fb7dd645f7cdabb0114bd4"} Jul 12 18:51:02.415560 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.415534 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:51:02.505764 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.505727 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-workload-certs\") pod \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " Jul 12 18:51:02.505926 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.505777 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-workload-socket\") pod \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " Jul 12 18:51:02.505926 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.505841 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-envoy\") pod \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " Jul 12 18:51:02.505926 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.505889 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-credential-socket\") pod \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " Jul 12 18:51:02.505926 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.505923 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-token\") pod \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " Jul 12 18:51:02.506130 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.505959 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-podinfo\") pod \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " Jul 12 18:51:02.506130 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.505990 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjwhn\" (UniqueName: \"kubernetes.io/projected/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-kube-api-access-fjwhn\") pod \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " Jul 12 18:51:02.506130 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.506068 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istiod-ca-cert\") pod \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " Jul 12 18:51:02.506130 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.506072 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" (UID: "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:51:02.506334 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.506113 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" (UID: "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:51:02.506334 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.506134 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-data\") pod \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\" (UID: \"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed\") " Jul 12 18:51:02.506334 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.506188 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" (UID: "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:51:02.506543 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.506494 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" (UID: "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:51:02.506611 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.506539 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-workload-certs\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:51:02.506611 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.506562 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-workload-socket\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:51:02.506611 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.506579 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-credential-socket\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:51:02.507124 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.507101 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-data" (OuterVolumeSpecName: "istio-data") pod "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" (UID: "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:51:02.508480 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.508449 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" (UID: "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 18:51:02.508598 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.508580 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-kube-api-access-fjwhn" (OuterVolumeSpecName: "kube-api-access-fjwhn") pod "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" (UID: "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed"). InnerVolumeSpecName "kube-api-access-fjwhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:51:02.508673 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.508653 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" (UID: "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:51:02.508673 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.508665 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-token" (OuterVolumeSpecName: "istio-token") pod "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" (UID: "1aac6669-afc7-4ca4-a3e5-afe457f4b9ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:51:02.607130 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.607042 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istiod-ca-cert\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:51:02.607130 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.607074 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-data\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:51:02.607130 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.607082 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-envoy\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:51:02.607130 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.607090 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-token\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:51:02.607130 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.607098 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-istio-podinfo\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:51:02.607130 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:02.607106 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjwhn\" (UniqueName: \"kubernetes.io/projected/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed-kube-api-access-fjwhn\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:51:03.079793 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:03.079758 2576 generic.go:356] "Generic (PLEG): container finished" podID="1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" containerID="9890f9f03b11dc6fce712b43d38546629e63b82ef9ed62a80b981b472749715d" exitCode=0 Jul 12 18:51:03.080021 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:03.079852 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" Jul 12 18:51:03.080021 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:03.079858 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" event={"ID":"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed","Type":"ContainerDied","Data":"9890f9f03b11dc6fce712b43d38546629e63b82ef9ed62a80b981b472749715d"} Jul 12 18:51:03.080021 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:03.079915 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps" event={"ID":"1aac6669-afc7-4ca4-a3e5-afe457f4b9ed","Type":"ContainerDied","Data":"00125224b3e8d5de509566c9cd93d0c2f6f77450ef7e40b62677379979c8898c"} Jul 12 18:51:03.080021 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:03.079942 2576 scope.go:117] "RemoveContainer" containerID="9890f9f03b11dc6fce712b43d38546629e63b82ef9ed62a80b981b472749715d" Jul 12 18:51:03.081429 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:03.081406 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" event={"ID":"391e7e64-1f08-4696-90d6-71af43c4fe97","Type":"ContainerStarted","Data":"29e12497ef579d583bba0039e803c875b66f62d271ed816b4d680151cbdca5c5"} Jul 12 18:51:03.081556 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:03.081539 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" Jul 12 18:51:03.089282 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:03.089267 2576 scope.go:117] "RemoveContainer" containerID="adbaf4b2c0c55a58e2a8fa61750fa0afe7e86fb5841f83ec8e62ecf55df8dd0e" Jul 12 18:51:03.095913 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:03.095888 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps"] Jul 12 18:51:03.098591 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:03.098573 2576 scope.go:117] "RemoveContainer" containerID="9890f9f03b11dc6fce712b43d38546629e63b82ef9ed62a80b981b472749715d" Jul 12 18:51:03.098887 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:51:03.098864 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9890f9f03b11dc6fce712b43d38546629e63b82ef9ed62a80b981b472749715d\": container with ID starting with 9890f9f03b11dc6fce712b43d38546629e63b82ef9ed62a80b981b472749715d not found: ID does not exist" containerID="9890f9f03b11dc6fce712b43d38546629e63b82ef9ed62a80b981b472749715d" Jul 12 18:51:03.098967 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:03.098897 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9890f9f03b11dc6fce712b43d38546629e63b82ef9ed62a80b981b472749715d"} err="failed to get container status \"9890f9f03b11dc6fce712b43d38546629e63b82ef9ed62a80b981b472749715d\": rpc error: code = NotFound desc = could not find container \"9890f9f03b11dc6fce712b43d38546629e63b82ef9ed62a80b981b472749715d\": container with ID starting with 9890f9f03b11dc6fce712b43d38546629e63b82ef9ed62a80b981b472749715d not found: ID does not exist" Jul 12 18:51:03.098967 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:03.098916 2576 scope.go:117] "RemoveContainer" containerID="adbaf4b2c0c55a58e2a8fa61750fa0afe7e86fb5841f83ec8e62ecf55df8dd0e" Jul 12 18:51:03.099089 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:03.099070 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-443d18-openshift-default-777bdb86b5-vtfps"] Jul 12 18:51:03.099210 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:51:03.099190 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adbaf4b2c0c55a58e2a8fa61750fa0afe7e86fb5841f83ec8e62ecf55df8dd0e\": container with ID starting with adbaf4b2c0c55a58e2a8fa61750fa0afe7e86fb5841f83ec8e62ecf55df8dd0e not found: ID does not exist" containerID="adbaf4b2c0c55a58e2a8fa61750fa0afe7e86fb5841f83ec8e62ecf55df8dd0e" Jul 12 18:51:03.099249 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:03.099217 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adbaf4b2c0c55a58e2a8fa61750fa0afe7e86fb5841f83ec8e62ecf55df8dd0e"} err="failed to get container status \"adbaf4b2c0c55a58e2a8fa61750fa0afe7e86fb5841f83ec8e62ecf55df8dd0e\": rpc error: code = NotFound desc = could not find container \"adbaf4b2c0c55a58e2a8fa61750fa0afe7e86fb5841f83ec8e62ecf55df8dd0e\": container with ID starting with adbaf4b2c0c55a58e2a8fa61750fa0afe7e86fb5841f83ec8e62ecf55df8dd0e not found: ID does not exist" Jul 12 18:51:03.111744 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:03.111696 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" podStartSLOduration=1.6723473960000002 podStartE2EDuration="2.111679688s" totalImagesPullingTime="439.332292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:51:03.110109736 +0000 UTC m=+2024.877407318" watchObservedRunningTime="2026-07-12 18:51:03.111679688 +0000 UTC m=+2024.878977269" Jul 12 18:51:04.817874 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:04.817833 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" path="/var/lib/kubelet/pods/1aac6669-afc7-4ca4-a3e5-afe457f4b9ed/volumes" Jul 12 18:51:05.667006 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:05.666968 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:05.980510 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:05.980405 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:06.235658 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.235565 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj"] Jul 12 18:51:06.236250 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.236233 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" containerName="istio-proxy" Jul 12 18:51:06.236301 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.236254 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" containerName="istio-proxy" Jul 12 18:51:06.236381 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.236370 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" containerName="istio-proxy" Jul 12 18:51:06.241498 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.241474 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.244120 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.244095 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-1f3102-openshift-default-dockercfg-4lmb7\"" Jul 12 18:51:06.250248 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.250225 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj"] Jul 12 18:51:06.260471 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.260434 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:06.342134 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.342100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-token\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.342340 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.342153 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mszhf\" (UniqueName: \"kubernetes.io/projected/d59f0898-2e95-4315-b73c-f5a4e72d9631-kube-api-access-mszhf\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.342340 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.342219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d59f0898-2e95-4315-b73c-f5a4e72d9631-istiod-ca-cert\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.342340 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.342275 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-workload-socket\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.342340 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.342314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-credential-socket\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.342553 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.342353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-data\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.342553 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.342417 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-envoy\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.342553 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.342471 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-podinfo\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.342553 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.342522 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-workload-certs\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.444162 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.444112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-envoy\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.444162 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.444164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-podinfo\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.444429 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.444194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-workload-certs\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.444429 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.444251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-token\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.444429 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.444298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mszhf\" (UniqueName: \"kubernetes.io/projected/d59f0898-2e95-4315-b73c-f5a4e72d9631-kube-api-access-mszhf\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.444429 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.444333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d59f0898-2e95-4315-b73c-f5a4e72d9631-istiod-ca-cert\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.444429 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.444381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-workload-socket\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.444429 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.444408 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-credential-socket\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.444728 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.444443 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-data\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.444728 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.444629 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-workload-certs\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.444860 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.444849 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-workload-socket\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.444922 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.444892 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-data\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.445039 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.445018 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-credential-socket\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.445357 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.445334 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d59f0898-2e95-4315-b73c-f5a4e72d9631-istiod-ca-cert\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.446678 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.446651 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-envoy\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.446952 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.446929 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-podinfo\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.451753 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.451726 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-token\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.452218 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.452203 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mszhf\" (UniqueName: \"kubernetes.io/projected/d59f0898-2e95-4315-b73c-f5a4e72d9631-kube-api-access-mszhf\") pod \"e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.553208 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.553116 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:06.608632 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.608586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:06.698077 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.698046 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj"] Jul 12 18:51:06.699802 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:51:06.699763 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59f0898_2e95_4315_b73c_f5a4e72d9631.slice/crio-c6de34bc460910a5bc1389d03039fa1bc2fbd4a066dc325a0a0f67f135861217 WatchSource:0}: Error finding container c6de34bc460910a5bc1389d03039fa1bc2fbd4a066dc325a0a0f67f135861217: Status 404 returned error can't find the container with id c6de34bc460910a5bc1389d03039fa1bc2fbd4a066dc325a0a0f67f135861217 Jul 12 18:51:06.702429 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.702394 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:51:06.702522 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.702462 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:51:06.702522 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:06.702493 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:51:07.103290 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:07.103253 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" event={"ID":"d59f0898-2e95-4315-b73c-f5a4e72d9631","Type":"ContainerStarted","Data":"a2c19f0290794f691e93b476888c4affe7c3af5966711e6026d85108fc41f140"} Jul 12 18:51:07.103684 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:07.103321 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" event={"ID":"d59f0898-2e95-4315-b73c-f5a4e72d9631","Type":"ContainerStarted","Data":"c6de34bc460910a5bc1389d03039fa1bc2fbd4a066dc325a0a0f67f135861217"} Jul 12 18:51:07.123763 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:07.123714 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" podStartSLOduration=1.123698628 podStartE2EDuration="1.123698628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:51:07.120944063 +0000 UTC m=+2028.888241645" watchObservedRunningTime="2026-07-12 18:51:07.123698628 +0000 UTC m=+2028.890996209" Jul 12 18:51:07.554013 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:07.553927 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:07.558920 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:07.558893 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:08.108654 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:08.108621 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:08.109706 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:08.109682 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:08.976713 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:08.976668 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:09.044726 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.044674 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:09.046089 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.045989 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5"] Jul 12 18:51:09.046552 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.046534 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" containerName="istio-proxy" Jul 12 18:51:09.046617 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.046553 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" containerName="istio-proxy" Jul 12 18:51:09.046669 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.046648 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aac6669-afc7-4ca4-a3e5-afe457f4b9ed" containerName="istio-proxy" Jul 12 18:51:09.050003 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.049988 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" Jul 12 18:51:09.052340 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.052314 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-1f3102-serving-cert\"" Jul 12 18:51:09.059211 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.058621 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5"] Jul 12 18:51:09.093048 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.093016 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" Jul 12 18:51:09.110798 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.110761 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:09.174022 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.173981 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51bcd823-2d74-4801-a073-30f12788f8b0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5\" (UID: \"51bcd823-2d74-4801-a073-30f12788f8b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" Jul 12 18:51:09.174198 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.174027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51bcd823-2d74-4801-a073-30f12788f8b0-maas-api-tls\") pod \"maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5\" (UID: \"51bcd823-2d74-4801-a073-30f12788f8b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" Jul 12 18:51:09.174198 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.174054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zwzx\" (UniqueName: \"kubernetes.io/projected/51bcd823-2d74-4801-a073-30f12788f8b0-kube-api-access-4zwzx\") pod \"maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5\" (UID: \"51bcd823-2d74-4801-a073-30f12788f8b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" Jul 12 18:51:09.259723 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.259630 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:09.274897 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.274858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51bcd823-2d74-4801-a073-30f12788f8b0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5\" (UID: \"51bcd823-2d74-4801-a073-30f12788f8b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" Jul 12 18:51:09.275079 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.274914 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51bcd823-2d74-4801-a073-30f12788f8b0-maas-api-tls\") pod \"maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5\" (UID: \"51bcd823-2d74-4801-a073-30f12788f8b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" Jul 12 18:51:09.275079 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.274945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zwzx\" (UniqueName: \"kubernetes.io/projected/51bcd823-2d74-4801-a073-30f12788f8b0-kube-api-access-4zwzx\") pod \"maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5\" (UID: \"51bcd823-2d74-4801-a073-30f12788f8b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" Jul 12 18:51:09.275614 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.275589 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51bcd823-2d74-4801-a073-30f12788f8b0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5\" (UID: \"51bcd823-2d74-4801-a073-30f12788f8b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" Jul 12 18:51:09.277594 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.277575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51bcd823-2d74-4801-a073-30f12788f8b0-maas-api-tls\") pod \"maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5\" (UID: \"51bcd823-2d74-4801-a073-30f12788f8b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" Jul 12 18:51:09.283339 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.283313 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zwzx\" (UniqueName: \"kubernetes.io/projected/51bcd823-2d74-4801-a073-30f12788f8b0-kube-api-access-4zwzx\") pod \"maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5\" (UID: \"51bcd823-2d74-4801-a073-30f12788f8b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" Jul 12 18:51:09.363866 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.363812 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" Jul 12 18:51:09.517221 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.517193 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5"] Jul 12 18:51:09.519398 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:51:09.519370 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51bcd823_2d74_4801_a073_30f12788f8b0.slice/crio-c45e1b15c6a615f593bf8bbf7b0728d0ad4ee45413ac6cf2fab0d64f7fe2ce1e WatchSource:0}: Error finding container c45e1b15c6a615f593bf8bbf7b0728d0ad4ee45413ac6cf2fab0d64f7fe2ce1e: Status 404 returned error can't find the container with id c45e1b15c6a615f593bf8bbf7b0728d0ad4ee45413ac6cf2fab0d64f7fe2ce1e Jul 12 18:51:09.539361 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.539326 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:09.637488 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:09.637452 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:10.123443 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:10.123406 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" event={"ID":"51bcd823-2d74-4801-a073-30f12788f8b0","Type":"ContainerStarted","Data":"f74f606ef3cd8a00d7ad827a6ab8e525633b1c2778d43f10f779e79943ae5065"} Jul 12 18:51:10.123863 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:10.123462 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" event={"ID":"51bcd823-2d74-4801-a073-30f12788f8b0","Type":"ContainerStarted","Data":"c45e1b15c6a615f593bf8bbf7b0728d0ad4ee45413ac6cf2fab0d64f7fe2ce1e"} Jul 12 18:51:10.123863 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:10.123715 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" Jul 12 18:51:10.139559 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:10.139492 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" podStartSLOduration=0.68453028 podStartE2EDuration="1.139470464s" totalImagesPullingTime="454.940184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 18:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:51:10.138446001 +0000 UTC m=+2031.905743603" watchObservedRunningTime="2026-07-12 18:51:10.139470464 +0000 UTC m=+2031.906768046" Jul 12 18:51:13.486686 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:13.486650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:16.135926 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:16.135893 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" Jul 12 18:51:22.406229 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:22.406195 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:22.506667 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:22.506616 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:24.085508 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:24.085467 2576 patch_prober.go:28] interesting pod/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": read tcp 10.134.0.2:36290->10.134.0.63:15021: read: connection reset by peer" start-of-body= Jul 12 18:51:24.085898 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:24.085539 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" podUID="d59f0898-2e95-4315-b73c-f5a4e72d9631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": read tcp 10.134.0.2:36290->10.134.0.63:15021: read: connection reset by peer" Jul 12 18:51:24.200501 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:24.200462 2576 generic.go:356] "Generic (PLEG): container finished" podID="d59f0898-2e95-4315-b73c-f5a4e72d9631" containerID="a2c19f0290794f691e93b476888c4affe7c3af5966711e6026d85108fc41f140" exitCode=0 Jul 12 18:51:24.200669 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:24.200532 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" event={"ID":"d59f0898-2e95-4315-b73c-f5a4e72d9631","Type":"ContainerDied","Data":"a2c19f0290794f691e93b476888c4affe7c3af5966711e6026d85108fc41f140"} Jul 12 18:51:24.200995 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:24.200980 2576 scope.go:117] "RemoveContainer" containerID="a2c19f0290794f691e93b476888c4affe7c3af5966711e6026d85108fc41f140" Jul 12 18:51:24.202070 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:24.202041 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:51:24.202154 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:24.202140 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:51:24.202198 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:24.202175 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 18:51:25.207243 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:25.207204 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" event={"ID":"d59f0898-2e95-4315-b73c-f5a4e72d9631","Type":"ContainerStarted","Data":"e6f90e28fe120718d3590d39ff3c360614b1e9c0b92ae293591281584e04f4f2"} Jul 12 18:51:25.553895 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:25.553796 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:25.554089 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:25.554062 2576 patch_prober.go:28] interesting pod/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 12 18:51:25.554158 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:25.554134 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" podUID="d59f0898-2e95-4315-b73c-f5a4e72d9631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 12 18:51:26.554556 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:26.554512 2576 patch_prober.go:28] interesting pod/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 12 18:51:26.555025 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:26.554583 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" podUID="d59f0898-2e95-4315-b73c-f5a4e72d9631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 12 18:51:27.553940 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:27.553901 2576 patch_prober.go:28] interesting pod/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 12 18:51:27.554156 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:27.553967 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" podUID="d59f0898-2e95-4315-b73c-f5a4e72d9631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 12 18:51:28.554246 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:28.554203 2576 patch_prober.go:28] interesting pod/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 12 18:51:28.554630 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:28.554275 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" podUID="d59f0898-2e95-4315-b73c-f5a4e72d9631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 12 18:51:29.558295 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:29.558261 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:29.558720 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:29.558540 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:29.559371 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:29.559354 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:51:30.996340 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:30.996301 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:51:31.097528 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:51:31.097496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:52:25.562508 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:52:25.562468 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:52:25.663458 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:52:25.663423 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:52:27.620971 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:52:27.620941 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:52:27.672961 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:52:27.672924 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:52:31.300261 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:52:31.300230 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:53:09.063946 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:53:09.063909 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:54:18.298797 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:18.298744 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:54:18.930457 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:18.930425 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh"] Jul 12 18:54:18.930666 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:18.930623 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" podUID="391e7e64-1f08-4696-90d6-71af43c4fe97" containerName="maas-api" containerID="cri-o://29e12497ef579d583bba0039e803c875b66f62d271ed816b4d680151cbdca5c5" gracePeriod=30 Jul 12 18:54:19.088704 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:19.088659 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" podUID="391e7e64-1f08-4696-90d6-71af43c4fe97" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.62:8443/health\": dial tcp 10.134.0.62:8443: connect: connection refused" Jul 12 18:54:19.111214 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:19.111176 2576 generic.go:356] "Generic (PLEG): container finished" podID="391e7e64-1f08-4696-90d6-71af43c4fe97" containerID="29e12497ef579d583bba0039e803c875b66f62d271ed816b4d680151cbdca5c5" exitCode=0 Jul 12 18:54:19.111356 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:19.111250 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" event={"ID":"391e7e64-1f08-4696-90d6-71af43c4fe97","Type":"ContainerDied","Data":"29e12497ef579d583bba0039e803c875b66f62d271ed816b4d680151cbdca5c5"} Jul 12 18:54:19.195238 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:19.195172 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" Jul 12 18:54:19.306654 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:19.306620 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/391e7e64-1f08-4696-90d6-71af43c4fe97-openshift-service-ca\") pod \"391e7e64-1f08-4696-90d6-71af43c4fe97\" (UID: \"391e7e64-1f08-4696-90d6-71af43c4fe97\") " Jul 12 18:54:19.306654 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:19.306662 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdsf9\" (UniqueName: \"kubernetes.io/projected/391e7e64-1f08-4696-90d6-71af43c4fe97-kube-api-access-sdsf9\") pod \"391e7e64-1f08-4696-90d6-71af43c4fe97\" (UID: \"391e7e64-1f08-4696-90d6-71af43c4fe97\") " Jul 12 18:54:19.307180 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:19.306702 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/391e7e64-1f08-4696-90d6-71af43c4fe97-maas-api-tls\") pod \"391e7e64-1f08-4696-90d6-71af43c4fe97\" (UID: \"391e7e64-1f08-4696-90d6-71af43c4fe97\") " Jul 12 18:54:19.307180 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:19.307017 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391e7e64-1f08-4696-90d6-71af43c4fe97-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "391e7e64-1f08-4696-90d6-71af43c4fe97" (UID: "391e7e64-1f08-4696-90d6-71af43c4fe97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:54:19.308863 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:19.308839 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391e7e64-1f08-4696-90d6-71af43c4fe97-kube-api-access-sdsf9" (OuterVolumeSpecName: "kube-api-access-sdsf9") pod "391e7e64-1f08-4696-90d6-71af43c4fe97" (UID: "391e7e64-1f08-4696-90d6-71af43c4fe97"). InnerVolumeSpecName "kube-api-access-sdsf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:54:19.308960 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:19.308864 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391e7e64-1f08-4696-90d6-71af43c4fe97-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "391e7e64-1f08-4696-90d6-71af43c4fe97" (UID: "391e7e64-1f08-4696-90d6-71af43c4fe97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:54:19.407737 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:19.407698 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/391e7e64-1f08-4696-90d6-71af43c4fe97-openshift-service-ca\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:54:19.407737 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:19.407726 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdsf9\" (UniqueName: \"kubernetes.io/projected/391e7e64-1f08-4696-90d6-71af43c4fe97-kube-api-access-sdsf9\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:54:19.407737 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:19.407737 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/391e7e64-1f08-4696-90d6-71af43c4fe97-maas-api-tls\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:54:19.448501 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:19.448392 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:54:20.118430 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:20.118399 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" Jul 12 18:54:20.118636 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:20.118395 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh" event={"ID":"391e7e64-1f08-4696-90d6-71af43c4fe97","Type":"ContainerDied","Data":"1c0ecedca46386125d3c09628fbb1b94b36278a3a8fb7dd645f7cdabb0114bd4"} Jul 12 18:54:20.118636 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:20.118537 2576 scope.go:117] "RemoveContainer" containerID="29e12497ef579d583bba0039e803c875b66f62d271ed816b4d680151cbdca5c5" Jul 12 18:54:20.140347 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:20.140322 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh"] Jul 12 18:54:20.144239 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:20.144213 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1a89be-56867bb5df-hmxjh"] Jul 12 18:54:20.819004 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:20.818967 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="391e7e64-1f08-4696-90d6-71af43c4fe97" path="/var/lib/kubelet/pods/391e7e64-1f08-4696-90d6-71af43c4fe97/volumes" Jul 12 18:54:24.378058 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:24.378019 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:54:24.480425 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:24.480392 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:54:24.631331 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:24.631241 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:54:24.997734 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:24.997648 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:54:25.405922 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:25.405882 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:54:25.509110 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:25.509070 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:54:25.610184 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:25.610151 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:54:25.730713 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:25.730626 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5"] Jul 12 18:54:25.730985 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:25.730944 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" podUID="51bcd823-2d74-4801-a073-30f12788f8b0" containerName="maas-api" containerID="cri-o://f74f606ef3cd8a00d7ad827a6ab8e525633b1c2778d43f10f779e79943ae5065" gracePeriod=30 Jul 12 18:54:25.751414 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:25.751388 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:54:25.988804 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:25.988730 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" Jul 12 18:54:26.155904 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.155871 2576 generic.go:356] "Generic (PLEG): container finished" podID="51bcd823-2d74-4801-a073-30f12788f8b0" containerID="f74f606ef3cd8a00d7ad827a6ab8e525633b1c2778d43f10f779e79943ae5065" exitCode=0 Jul 12 18:54:26.156087 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.155910 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" Jul 12 18:54:26.156087 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.155950 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" event={"ID":"51bcd823-2d74-4801-a073-30f12788f8b0","Type":"ContainerDied","Data":"f74f606ef3cd8a00d7ad827a6ab8e525633b1c2778d43f10f779e79943ae5065"} Jul 12 18:54:26.156087 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.156002 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5" event={"ID":"51bcd823-2d74-4801-a073-30f12788f8b0","Type":"ContainerDied","Data":"c45e1b15c6a615f593bf8bbf7b0728d0ad4ee45413ac6cf2fab0d64f7fe2ce1e"} Jul 12 18:54:26.156087 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.156027 2576 scope.go:117] "RemoveContainer" containerID="f74f606ef3cd8a00d7ad827a6ab8e525633b1c2778d43f10f779e79943ae5065" Jul 12 18:54:26.164675 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.164653 2576 scope.go:117] "RemoveContainer" containerID="f74f606ef3cd8a00d7ad827a6ab8e525633b1c2778d43f10f779e79943ae5065" Jul 12 18:54:26.164981 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:54:26.164962 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f74f606ef3cd8a00d7ad827a6ab8e525633b1c2778d43f10f779e79943ae5065\": container with ID starting with f74f606ef3cd8a00d7ad827a6ab8e525633b1c2778d43f10f779e79943ae5065 not found: ID does not exist" containerID="f74f606ef3cd8a00d7ad827a6ab8e525633b1c2778d43f10f779e79943ae5065" Jul 12 18:54:26.165040 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.164990 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f74f606ef3cd8a00d7ad827a6ab8e525633b1c2778d43f10f779e79943ae5065"} err="failed to get container status \"f74f606ef3cd8a00d7ad827a6ab8e525633b1c2778d43f10f779e79943ae5065\": rpc error: code = NotFound desc = could not find container \"f74f606ef3cd8a00d7ad827a6ab8e525633b1c2778d43f10f779e79943ae5065\": container with ID starting with f74f606ef3cd8a00d7ad827a6ab8e525633b1c2778d43f10f779e79943ae5065 not found: ID does not exist" Jul 12 18:54:26.170298 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.170276 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51bcd823-2d74-4801-a073-30f12788f8b0-openshift-service-ca\") pod \"51bcd823-2d74-4801-a073-30f12788f8b0\" (UID: \"51bcd823-2d74-4801-a073-30f12788f8b0\") " Jul 12 18:54:26.170392 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.170316 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zwzx\" (UniqueName: \"kubernetes.io/projected/51bcd823-2d74-4801-a073-30f12788f8b0-kube-api-access-4zwzx\") pod \"51bcd823-2d74-4801-a073-30f12788f8b0\" (UID: \"51bcd823-2d74-4801-a073-30f12788f8b0\") " Jul 12 18:54:26.170455 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.170441 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51bcd823-2d74-4801-a073-30f12788f8b0-maas-api-tls\") pod \"51bcd823-2d74-4801-a073-30f12788f8b0\" (UID: \"51bcd823-2d74-4801-a073-30f12788f8b0\") " Jul 12 18:54:26.170665 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.170634 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51bcd823-2d74-4801-a073-30f12788f8b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "51bcd823-2d74-4801-a073-30f12788f8b0" (UID: "51bcd823-2d74-4801-a073-30f12788f8b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:54:26.170827 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.170805 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51bcd823-2d74-4801-a073-30f12788f8b0-openshift-service-ca\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:54:26.172413 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.172389 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51bcd823-2d74-4801-a073-30f12788f8b0-kube-api-access-4zwzx" (OuterVolumeSpecName: "kube-api-access-4zwzx") pod "51bcd823-2d74-4801-a073-30f12788f8b0" (UID: "51bcd823-2d74-4801-a073-30f12788f8b0"). InnerVolumeSpecName "kube-api-access-4zwzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:54:26.172471 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.172454 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51bcd823-2d74-4801-a073-30f12788f8b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "51bcd823-2d74-4801-a073-30f12788f8b0" (UID: "51bcd823-2d74-4801-a073-30f12788f8b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 18:54:26.271580 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.271495 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zwzx\" (UniqueName: \"kubernetes.io/projected/51bcd823-2d74-4801-a073-30f12788f8b0-kube-api-access-4zwzx\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:54:26.271580 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.271528 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51bcd823-2d74-4801-a073-30f12788f8b0-maas-api-tls\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:54:26.477418 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.477386 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5"] Jul 12 18:54:26.480525 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.480500 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f3102-75d595d7cf-xr6v5"] Jul 12 18:54:26.818642 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:26.818612 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51bcd823-2d74-4801-a073-30f12788f8b0" path="/var/lib/kubelet/pods/51bcd823-2d74-4801-a073-30f12788f8b0/volumes" Jul 12 18:54:31.172059 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:31.172019 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:54:31.271493 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:31.271456 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:54:31.388601 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:31.388569 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj"] Jul 12 18:54:31.388836 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:31.388810 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" podUID="d59f0898-2e95-4315-b73c-f5a4e72d9631" containerName="istio-proxy" containerID="cri-o://e6f90e28fe120718d3590d39ff3c360614b1e9c0b92ae293591281584e04f4f2" gracePeriod=30 Jul 12 18:54:31.409639 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:31.409609 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:54:32.171802 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:32.171751 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7ccdcbfdd6-8lv2s_4d715528-b0d2-4eed-b118-396125ab82c7/authorino/0.log" Jul 12 18:54:36.578224 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.578190 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-z2545_e7b07d4c-cecc-49b7-8480-24163ba3fdb6/manager/0.log" Jul 12 18:54:36.654156 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.654132 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:54:36.669615 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.669549 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-workload-socket\") pod \"d59f0898-2e95-4315-b73c-f5a4e72d9631\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " Jul 12 18:54:36.669737 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.669620 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-credential-socket\") pod \"d59f0898-2e95-4315-b73c-f5a4e72d9631\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " Jul 12 18:54:36.669737 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.669668 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d59f0898-2e95-4315-b73c-f5a4e72d9631-istiod-ca-cert\") pod \"d59f0898-2e95-4315-b73c-f5a4e72d9631\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " Jul 12 18:54:36.669737 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.669714 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mszhf\" (UniqueName: \"kubernetes.io/projected/d59f0898-2e95-4315-b73c-f5a4e72d9631-kube-api-access-mszhf\") pod \"d59f0898-2e95-4315-b73c-f5a4e72d9631\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " Jul 12 18:54:36.669904 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.669754 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-workload-certs\") pod \"d59f0898-2e95-4315-b73c-f5a4e72d9631\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " Jul 12 18:54:36.669904 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.669808 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-token\") pod \"d59f0898-2e95-4315-b73c-f5a4e72d9631\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " Jul 12 18:54:36.669904 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.669853 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d59f0898-2e95-4315-b73c-f5a4e72d9631" (UID: "d59f0898-2e95-4315-b73c-f5a4e72d9631"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:54:36.669904 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.669877 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-data\") pod \"d59f0898-2e95-4315-b73c-f5a4e72d9631\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " Jul 12 18:54:36.669904 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.669902 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-envoy\") pod \"d59f0898-2e95-4315-b73c-f5a4e72d9631\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " Jul 12 18:54:36.670100 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.669936 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-podinfo\") pod \"d59f0898-2e95-4315-b73c-f5a4e72d9631\" (UID: \"d59f0898-2e95-4315-b73c-f5a4e72d9631\") " Jul 12 18:54:36.670100 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.669955 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d59f0898-2e95-4315-b73c-f5a4e72d9631" (UID: "d59f0898-2e95-4315-b73c-f5a4e72d9631"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:54:36.670191 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.670170 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59f0898-2e95-4315-b73c-f5a4e72d9631-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d59f0898-2e95-4315-b73c-f5a4e72d9631" (UID: "d59f0898-2e95-4315-b73c-f5a4e72d9631"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 18:54:36.670290 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.670266 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-credential-socket\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:54:36.670356 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.670278 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d59f0898-2e95-4315-b73c-f5a4e72d9631" (UID: "d59f0898-2e95-4315-b73c-f5a4e72d9631"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:54:36.670356 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.670296 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d59f0898-2e95-4315-b73c-f5a4e72d9631-istiod-ca-cert\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:54:36.670356 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.670311 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-workload-socket\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:54:36.670742 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.670713 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-data" (OuterVolumeSpecName: "istio-data") pod "d59f0898-2e95-4315-b73c-f5a4e72d9631" (UID: "d59f0898-2e95-4315-b73c-f5a4e72d9631"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:54:36.672545 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.672339 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d59f0898-2e95-4315-b73c-f5a4e72d9631" (UID: "d59f0898-2e95-4315-b73c-f5a4e72d9631"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 18:54:36.672756 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.672732 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d59f0898-2e95-4315-b73c-f5a4e72d9631" (UID: "d59f0898-2e95-4315-b73c-f5a4e72d9631"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 18:54:36.673152 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.673131 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-token" (OuterVolumeSpecName: "istio-token") pod "d59f0898-2e95-4315-b73c-f5a4e72d9631" (UID: "d59f0898-2e95-4315-b73c-f5a4e72d9631"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:54:36.673225 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.673163 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59f0898-2e95-4315-b73c-f5a4e72d9631-kube-api-access-mszhf" (OuterVolumeSpecName: "kube-api-access-mszhf") pod "d59f0898-2e95-4315-b73c-f5a4e72d9631" (UID: "d59f0898-2e95-4315-b73c-f5a4e72d9631"). InnerVolumeSpecName "kube-api-access-mszhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 18:54:36.696994 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.696938 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-c9sjr_25619e89-db09-4b28-b63c-1b5e432c719c/manager/0.log" Jul 12 18:54:36.771230 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.771189 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-token\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:54:36.771230 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.771223 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-data\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:54:36.771230 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.771234 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-envoy\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:54:36.771469 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.771246 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d59f0898-2e95-4315-b73c-f5a4e72d9631-istio-podinfo\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:54:36.771469 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.771257 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mszhf\" (UniqueName: \"kubernetes.io/projected/d59f0898-2e95-4315-b73c-f5a4e72d9631-kube-api-access-mszhf\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:54:36.771469 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.771269 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d59f0898-2e95-4315-b73c-f5a4e72d9631-workload-certs\") on node \"ip-10-0-132-5.ec2.internal\" DevicePath \"\"" Jul 12 18:54:36.811435 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.811403 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bcb555768-r7d65_4751ce85-56bc-4992-a7a8-ba1543377dc8/manager/0.log" Jul 12 18:54:36.921051 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:36.920972 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2rln5_b72d8bae-1014-49ea-8f13-dff0f39c5ad2/server/0.log" Jul 12 18:54:37.214101 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:37.214012 2576 generic.go:356] "Generic (PLEG): container finished" podID="d59f0898-2e95-4315-b73c-f5a4e72d9631" containerID="e6f90e28fe120718d3590d39ff3c360614b1e9c0b92ae293591281584e04f4f2" exitCode=0 Jul 12 18:54:37.214101 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:37.214061 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" Jul 12 18:54:37.214101 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:37.214082 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" event={"ID":"d59f0898-2e95-4315-b73c-f5a4e72d9631","Type":"ContainerDied","Data":"e6f90e28fe120718d3590d39ff3c360614b1e9c0b92ae293591281584e04f4f2"} Jul 12 18:54:37.214345 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:37.214124 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj" event={"ID":"d59f0898-2e95-4315-b73c-f5a4e72d9631","Type":"ContainerDied","Data":"c6de34bc460910a5bc1389d03039fa1bc2fbd4a066dc325a0a0f67f135861217"} Jul 12 18:54:37.214345 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:37.214146 2576 scope.go:117] "RemoveContainer" containerID="e6f90e28fe120718d3590d39ff3c360614b1e9c0b92ae293591281584e04f4f2" Jul 12 18:54:37.222812 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:37.222778 2576 scope.go:117] "RemoveContainer" containerID="a2c19f0290794f691e93b476888c4affe7c3af5966711e6026d85108fc41f140" Jul 12 18:54:37.231926 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:37.231904 2576 scope.go:117] "RemoveContainer" containerID="e6f90e28fe120718d3590d39ff3c360614b1e9c0b92ae293591281584e04f4f2" Jul 12 18:54:37.232205 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:54:37.232185 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f90e28fe120718d3590d39ff3c360614b1e9c0b92ae293591281584e04f4f2\": container with ID starting with e6f90e28fe120718d3590d39ff3c360614b1e9c0b92ae293591281584e04f4f2 not found: ID does not exist" containerID="e6f90e28fe120718d3590d39ff3c360614b1e9c0b92ae293591281584e04f4f2" Jul 12 18:54:37.232272 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:37.232215 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f90e28fe120718d3590d39ff3c360614b1e9c0b92ae293591281584e04f4f2"} err="failed to get container status \"e6f90e28fe120718d3590d39ff3c360614b1e9c0b92ae293591281584e04f4f2\": rpc error: code = NotFound desc = could not find container \"e6f90e28fe120718d3590d39ff3c360614b1e9c0b92ae293591281584e04f4f2\": container with ID starting with e6f90e28fe120718d3590d39ff3c360614b1e9c0b92ae293591281584e04f4f2 not found: ID does not exist" Jul 12 18:54:37.232272 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:37.232232 2576 scope.go:117] "RemoveContainer" containerID="a2c19f0290794f691e93b476888c4affe7c3af5966711e6026d85108fc41f140" Jul 12 18:54:37.232421 ip-10-0-132-5 kubenswrapper[2576]: E0712 18:54:37.232404 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c19f0290794f691e93b476888c4affe7c3af5966711e6026d85108fc41f140\": container with ID starting with a2c19f0290794f691e93b476888c4affe7c3af5966711e6026d85108fc41f140 not found: ID does not exist" containerID="a2c19f0290794f691e93b476888c4affe7c3af5966711e6026d85108fc41f140" Jul 12 18:54:37.232478 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:37.232424 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c19f0290794f691e93b476888c4affe7c3af5966711e6026d85108fc41f140"} err="failed to get container status \"a2c19f0290794f691e93b476888c4affe7c3af5966711e6026d85108fc41f140\": rpc error: code = NotFound desc = could not find container \"a2c19f0290794f691e93b476888c4affe7c3af5966711e6026d85108fc41f140\": container with ID starting with a2c19f0290794f691e93b476888c4affe7c3af5966711e6026d85108fc41f140 not found: ID does not exist" Jul 12 18:54:37.232478 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:37.232435 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj"] Jul 12 18:54:37.235431 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:37.235407 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-1f3102-openshift-default-5b4cd979f5-sgwhj"] Jul 12 18:54:37.405389 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:37.405364 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-db78fd4c7-w2mwd_9506c9b2-3061-445c-ac23-dd18dc76c38d/copy-manifests/0.log" Jul 12 18:54:37.417879 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:37.417851 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-db78fd4c7-w2mwd_9506c9b2-3061-445c-ac23-dd18dc76c38d/manager/0.log" Jul 12 18:54:38.644814 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:38.644768 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7ccdcbfdd6-8lv2s_4d715528-b0d2-4eed-b118-396125ab82c7/authorino/0.log" Jul 12 18:54:38.754979 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:38.754950 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-z9rk2_a1a031af-1057-4bfc-aac0-a49cb91c4226/manager/0.log" Jul 12 18:54:38.818394 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:38.818359 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d59f0898-2e95-4315-b73c-f5a4e72d9631" path="/var/lib/kubelet/pods/d59f0898-2e95-4315-b73c-f5a4e72d9631/volumes" Jul 12 18:54:38.868288 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:38.868241 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-mx5pz_035b263e-40a1-4103-945d-81b47513d05e/manager/0.log" Jul 12 18:54:38.971506 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:38.971430 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-xkn87_e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd/kuadrant-console-plugin/0.log" Jul 12 18:54:39.082931 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:39.082899 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-qb9vl_2321885c-3e61-4307-bd59-9244b38c26e3/registry-server/0.log" Jul 12 18:54:39.199266 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:39.199219 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-gzpxp_4679d403-b9fb-4cae-b5b3-9e297d5f8a5d/manager/0.log" Jul 12 18:54:39.421466 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:39.421437 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-rcdg6_00a1e377-7d3a-4796-90c9-75f34b4daf18/manager/0.log" Jul 12 18:54:40.090021 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:40.089982 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bd558f979-vxgtx_07bec7be-8262-4463-94e7-a69a40481a95/kube-auth-proxy/0.log" Jul 12 18:54:40.205647 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:40.205578 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tbk4x_678cd7d5-7144-400c-a44f-c3b07b36fab9/istio-proxy/1.log" Jul 12 18:54:40.316091 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:40.316059 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-9msj9_53951bfe-0dde-4ee0-9199-f2de0afb1c82/payload-pre-processing/0.log" Jul 12 18:54:40.427937 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:40.427861 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-v9qgm_629ea87f-db94-4c2c-ae5e-f1f48cb5d378/payload-processing/0.log" Jul 12 18:54:40.875577 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:40.875553 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-6qrbb_04f4958c-af58-421a-bcd7-1396d6f010d6/main/0.log" Jul 12 18:54:40.981989 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:40.981952 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-ztcll_c75a7175-8860-414c-980d-b22d57f898cd/main/0.log" Jul 12 18:54:45.290093 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:45.290064 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:54:45.387829 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:45.387766 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" limit=1024 Jul 12 18:54:48.037003 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:48.036973 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pjl7r_8a8808fa-07dc-4568-b10f-0a84be341957/global-pull-secret-syncer/0.log" Jul 12 18:54:48.078120 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:48.078089 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5fzsx_4e7716b6-9736-4301-a491-6d236a12ed7c/konnectivity-agent/0.log" Jul 12 18:54:48.159840 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:48.159810 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-5.ec2.internal_4642eca106af85a646d7b3db5a92c5a2/haproxy/0.log" Jul 12 18:54:53.302308 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:53.302226 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7ccdcbfdd6-8lv2s_4d715528-b0d2-4eed-b118-396125ab82c7/authorino/0.log" Jul 12 18:54:53.325855 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:53.325810 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-z9rk2_a1a031af-1057-4bfc-aac0-a49cb91c4226/manager/0.log" Jul 12 18:54:53.359687 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:53.359662 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-mx5pz_035b263e-40a1-4103-945d-81b47513d05e/manager/0.log" Jul 12 18:54:53.380482 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:53.380459 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-xkn87_e1b304e6-edfe-4e6c-9bc7-df2a3ad7acdd/kuadrant-console-plugin/0.log" Jul 12 18:54:53.417210 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:53.417181 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-qb9vl_2321885c-3e61-4307-bd59-9244b38c26e3/registry-server/0.log" Jul 12 18:54:53.521317 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:53.521282 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-gzpxp_4679d403-b9fb-4cae-b5b3-9e297d5f8a5d/manager/0.log" Jul 12 18:54:53.613714 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:53.613683 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-rcdg6_00a1e377-7d3a-4796-90c9-75f34b4daf18/manager/0.log" Jul 12 18:54:55.415694 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:55.415667 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-gvznh_4318b9e8-96d1-488f-8df6-8ebd16d3831d/monitoring-plugin/0.log" Jul 12 18:54:55.602764 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:55.602733 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vpp5s_9cbfbdb1-45ff-461a-9517-6174e7ce7dfa/node-exporter/0.log" Jul 12 18:54:55.623268 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:55.623240 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vpp5s_9cbfbdb1-45ff-461a-9517-6174e7ce7dfa/kube-rbac-proxy/0.log" Jul 12 18:54:55.641357 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:55.641331 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vpp5s_9cbfbdb1-45ff-461a-9517-6174e7ce7dfa/init-textfile/0.log" Jul 12 18:54:55.663729 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:55.663701 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-xmj8b_12e428fb-eb05-4442-90c5-183a117de7d5/kube-rbac-proxy-main/0.log" Jul 12 18:54:55.681500 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:55.681428 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-xmj8b_12e428fb-eb05-4442-90c5-183a117de7d5/kube-rbac-proxy-self/0.log" Jul 12 18:54:55.700360 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:55.700336 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-xmj8b_12e428fb-eb05-4442-90c5-183a117de7d5/openshift-state-metrics/0.log" Jul 12 18:54:55.737512 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:55.737480 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fea1ac3c-e030-4b05-a1aa-3b104ee07dc5/prometheus/0.log" Jul 12 18:54:55.763337 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:55.763307 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fea1ac3c-e030-4b05-a1aa-3b104ee07dc5/config-reloader/0.log" Jul 12 18:54:55.777073 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:55.777048 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fea1ac3c-e030-4b05-a1aa-3b104ee07dc5/thanos-sidecar/0.log" Jul 12 18:54:55.815553 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:55.815521 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fea1ac3c-e030-4b05-a1aa-3b104ee07dc5/kube-rbac-proxy-web/0.log" Jul 12 18:54:55.834239 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:55.834218 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fea1ac3c-e030-4b05-a1aa-3b104ee07dc5/kube-rbac-proxy/0.log" Jul 12 18:54:55.855234 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:55.855205 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fea1ac3c-e030-4b05-a1aa-3b104ee07dc5/kube-rbac-proxy-thanos/0.log" Jul 12 18:54:55.873576 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:55.873554 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fea1ac3c-e030-4b05-a1aa-3b104ee07dc5/init-config-reloader/0.log" Jul 12 18:54:55.898906 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:55.898880 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-pwrnr_eec26ff9-0bdb-42f3-ab2b-4deee347a4d9/prometheus-operator/0.log" Jul 12 18:54:55.922906 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:55.922880 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-pwrnr_eec26ff9-0bdb-42f3-ab2b-4deee347a4d9/kube-rbac-proxy/0.log" Jul 12 18:54:55.943770 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:55.943698 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-vths9_16e8851a-ed17-47d0-b6d4-f998ea699fc7/prometheus-operator-admission-webhook/0.log" Jul 12 18:54:56.039201 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:56.039176 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b9f97f8fc-xm6c6_f8c24501-1c77-4804-b703-5ae46ee76e6f/thanos-query/0.log" Jul 12 18:54:56.057284 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:56.057259 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b9f97f8fc-xm6c6_f8c24501-1c77-4804-b703-5ae46ee76e6f/kube-rbac-proxy-web/0.log" Jul 12 18:54:56.078540 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:56.078512 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b9f97f8fc-xm6c6_f8c24501-1c77-4804-b703-5ae46ee76e6f/kube-rbac-proxy/0.log" Jul 12 18:54:56.099499 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:56.099477 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b9f97f8fc-xm6c6_f8c24501-1c77-4804-b703-5ae46ee76e6f/prom-label-proxy/0.log" Jul 12 18:54:56.120128 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:56.120105 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b9f97f8fc-xm6c6_f8c24501-1c77-4804-b703-5ae46ee76e6f/kube-rbac-proxy-rules/0.log" Jul 12 18:54:56.137772 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:56.137744 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b9f97f8fc-xm6c6_f8c24501-1c77-4804-b703-5ae46ee76e6f/kube-rbac-proxy-metrics/0.log" Jul 12 18:54:57.519507 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.519470 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t"] Jul 12 18:54:57.519946 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.519922 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d59f0898-2e95-4315-b73c-f5a4e72d9631" containerName="istio-proxy" Jul 12 18:54:57.519946 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.519937 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d59f0898-2e95-4315-b73c-f5a4e72d9631" containerName="istio-proxy" Jul 12 18:54:57.520028 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.519956 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51bcd823-2d74-4801-a073-30f12788f8b0" containerName="maas-api" Jul 12 18:54:57.520028 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.519962 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="51bcd823-2d74-4801-a073-30f12788f8b0" containerName="maas-api" Jul 12 18:54:57.520028 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.519971 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d59f0898-2e95-4315-b73c-f5a4e72d9631" containerName="istio-proxy" Jul 12 18:54:57.520028 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.519976 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d59f0898-2e95-4315-b73c-f5a4e72d9631" containerName="istio-proxy" Jul 12 18:54:57.520028 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.519985 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="391e7e64-1f08-4696-90d6-71af43c4fe97" containerName="maas-api" Jul 12 18:54:57.520028 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.519991 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="391e7e64-1f08-4696-90d6-71af43c4fe97" containerName="maas-api" Jul 12 18:54:57.520288 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.520055 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d59f0898-2e95-4315-b73c-f5a4e72d9631" containerName="istio-proxy" Jul 12 18:54:57.520288 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.520067 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="391e7e64-1f08-4696-90d6-71af43c4fe97" containerName="maas-api" Jul 12 18:54:57.520288 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.520074 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="51bcd823-2d74-4801-a073-30f12788f8b0" containerName="maas-api" Jul 12 18:54:57.523216 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.523196 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:57.525673 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.525647 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lcvxn\"/\"kube-root-ca.crt\"" Jul 12 18:54:57.525768 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.525647 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lcvxn\"/\"default-dockercfg-flgp8\"" Jul 12 18:54:57.525768 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.525647 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lcvxn\"/\"openshift-service-ca.crt\"" Jul 12 18:54:57.532405 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.532381 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t"] Jul 12 18:54:57.651580 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.651544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0098c029-800a-4a80-9327-743972f8cc2a-podres\") pod \"perf-node-gather-daemonset-ztv4t\" (UID: \"0098c029-800a-4a80-9327-743972f8cc2a\") " pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:57.651580 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.651579 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjjrz\" (UniqueName: \"kubernetes.io/projected/0098c029-800a-4a80-9327-743972f8cc2a-kube-api-access-xjjrz\") pod \"perf-node-gather-daemonset-ztv4t\" (UID: \"0098c029-800a-4a80-9327-743972f8cc2a\") " pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:57.651829 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.651605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0098c029-800a-4a80-9327-743972f8cc2a-proc\") pod \"perf-node-gather-daemonset-ztv4t\" (UID: \"0098c029-800a-4a80-9327-743972f8cc2a\") " pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:57.651829 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.651626 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0098c029-800a-4a80-9327-743972f8cc2a-sys\") pod \"perf-node-gather-daemonset-ztv4t\" (UID: \"0098c029-800a-4a80-9327-743972f8cc2a\") " pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:57.651829 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.651693 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0098c029-800a-4a80-9327-743972f8cc2a-lib-modules\") pod \"perf-node-gather-daemonset-ztv4t\" (UID: \"0098c029-800a-4a80-9327-743972f8cc2a\") " pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:57.752551 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.752516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0098c029-800a-4a80-9327-743972f8cc2a-proc\") pod \"perf-node-gather-daemonset-ztv4t\" (UID: \"0098c029-800a-4a80-9327-743972f8cc2a\") " pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:57.752551 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.752552 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0098c029-800a-4a80-9327-743972f8cc2a-sys\") pod \"perf-node-gather-daemonset-ztv4t\" (UID: \"0098c029-800a-4a80-9327-743972f8cc2a\") " pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:57.752779 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.752606 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0098c029-800a-4a80-9327-743972f8cc2a-lib-modules\") pod \"perf-node-gather-daemonset-ztv4t\" (UID: \"0098c029-800a-4a80-9327-743972f8cc2a\") " pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:57.752779 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.752647 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0098c029-800a-4a80-9327-743972f8cc2a-proc\") pod \"perf-node-gather-daemonset-ztv4t\" (UID: \"0098c029-800a-4a80-9327-743972f8cc2a\") " pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:57.752779 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.752672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0098c029-800a-4a80-9327-743972f8cc2a-podres\") pod \"perf-node-gather-daemonset-ztv4t\" (UID: \"0098c029-800a-4a80-9327-743972f8cc2a\") " pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:57.752779 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.752699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjjrz\" (UniqueName: \"kubernetes.io/projected/0098c029-800a-4a80-9327-743972f8cc2a-kube-api-access-xjjrz\") pod \"perf-node-gather-daemonset-ztv4t\" (UID: \"0098c029-800a-4a80-9327-743972f8cc2a\") " pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:57.752779 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.752705 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0098c029-800a-4a80-9327-743972f8cc2a-sys\") pod \"perf-node-gather-daemonset-ztv4t\" (UID: \"0098c029-800a-4a80-9327-743972f8cc2a\") " pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:57.752978 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.752810 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0098c029-800a-4a80-9327-743972f8cc2a-lib-modules\") pod \"perf-node-gather-daemonset-ztv4t\" (UID: \"0098c029-800a-4a80-9327-743972f8cc2a\") " pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:57.752978 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.752819 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0098c029-800a-4a80-9327-743972f8cc2a-podres\") pod \"perf-node-gather-daemonset-ztv4t\" (UID: \"0098c029-800a-4a80-9327-743972f8cc2a\") " pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:57.760282 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.760260 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjjrz\" (UniqueName: \"kubernetes.io/projected/0098c029-800a-4a80-9327-743972f8cc2a-kube-api-access-xjjrz\") pod \"perf-node-gather-daemonset-ztv4t\" (UID: \"0098c029-800a-4a80-9327-743972f8cc2a\") " pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:57.834239 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.834210 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:57.965824 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:57.965798 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t"] Jul 12 18:54:57.967878 ip-10-0-132-5 kubenswrapper[2576]: W0712 18:54:57.967841 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0098c029_800a_4a80_9327_743972f8cc2a.slice/crio-819c7dfc2eea57c8d20d9651469e480e25e2c6094f94523d26ec9c3a5482f2ea WatchSource:0}: Error finding container 819c7dfc2eea57c8d20d9651469e480e25e2c6094f94523d26ec9c3a5482f2ea: Status 404 returned error can't find the container with id 819c7dfc2eea57c8d20d9651469e480e25e2c6094f94523d26ec9c3a5482f2ea Jul 12 18:54:58.318575 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:58.318489 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" event={"ID":"0098c029-800a-4a80-9327-743972f8cc2a","Type":"ContainerStarted","Data":"b51a6a1a43c213b4d75718b9b43794108ebf63dce408827258c7542fc60e7c45"} Jul 12 18:54:58.318575 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:58.318565 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" event={"ID":"0098c029-800a-4a80-9327-743972f8cc2a","Type":"ContainerStarted","Data":"819c7dfc2eea57c8d20d9651469e480e25e2c6094f94523d26ec9c3a5482f2ea"} Jul 12 18:54:58.318768 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:58.318601 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:54:58.333019 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:58.332974 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" podStartSLOduration=1.332959875 podStartE2EDuration="1.332959875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 18:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 18:54:58.332148988 +0000 UTC m=+2260.099446594" watchObservedRunningTime="2026-07-12 18:54:58.332959875 +0000 UTC m=+2260.100257451" Jul 12 18:54:58.408226 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:58.408196 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-zldlk_c6483f3c-7c4a-48d4-90f8-ff4deb9aa0a4/download-server/0.log" Jul 12 18:54:59.908884 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:59.908853 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wndcz_4df2da2e-9018-4a69-8776-f152cd0b6052/dns/0.log" Jul 12 18:54:59.925992 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:59.925963 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wndcz_4df2da2e-9018-4a69-8776-f152cd0b6052/kube-rbac-proxy/0.log" Jul 12 18:54:59.967062 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:54:59.967024 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rwd88_051fb076-c1be-4f39-bf98-7a0034d0e12e/dns-node-resolver/0.log" Jul 12 18:55:00.469530 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:00.469503 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5f788c444b-9sjcx_b554cd4b-708d-47ab-b6c2-76ae2587ee31/registry/0.log" Jul 12 18:55:00.507948 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:00.507919 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lpqkq_03a2383e-2c1a-4e4e-afbc-6fd5f678e702/node-ca/0.log" Jul 12 18:55:01.608074 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:01.608040 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bd558f979-vxgtx_07bec7be-8262-4463-94e7-a69a40481a95/kube-auth-proxy/0.log" Jul 12 18:55:01.633971 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:01.633935 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tbk4x_678cd7d5-7144-400c-a44f-c3b07b36fab9/istio-proxy/0.log" Jul 12 18:55:01.662852 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:01.662824 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tbk4x_678cd7d5-7144-400c-a44f-c3b07b36fab9/istio-proxy/1.log" Jul 12 18:55:01.701426 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:01.701396 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-9msj9_53951bfe-0dde-4ee0-9199-f2de0afb1c82/payload-pre-processing/0.log" Jul 12 18:55:01.743240 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:01.743212 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-v9qgm_629ea87f-db94-4c2c-ae5e-f1f48cb5d378/payload-processing/0.log" Jul 12 18:55:02.406873 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:02.406846 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c2zhg_70b7d9dd-771d-4d6c-be24-149e3fa53de0/serve-healthcheck-canary/0.log" Jul 12 18:55:02.961586 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:02.961552 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-88mc5_d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2/kube-rbac-proxy/0.log" Jul 12 18:55:02.979205 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:02.979183 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-88mc5_d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2/exporter/0.log" Jul 12 18:55:02.996880 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:02.996849 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-88mc5_d11eaf29-8c5a-405c-a9d0-1cb9c8a9dca2/extractor/0.log" Jul 12 18:55:04.333769 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:04.333740 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lcvxn/perf-node-gather-daemonset-ztv4t" Jul 12 18:55:05.047391 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:05.047358 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-z2545_e7b07d4c-cecc-49b7-8480-24163ba3fdb6/manager/0.log" Jul 12 18:55:05.147683 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:05.147650 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-c9sjr_25619e89-db09-4b28-b63c-1b5e432c719c/manager/0.log" Jul 12 18:55:05.229186 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:05.229151 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bcb555768-r7d65_4751ce85-56bc-4992-a7a8-ba1543377dc8/manager/0.log" Jul 12 18:55:05.247775 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:05.247751 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2rln5_b72d8bae-1014-49ea-8f13-dff0f39c5ad2/server/0.log" Jul 12 18:55:05.488878 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:05.488847 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-db78fd4c7-w2mwd_9506c9b2-3061-445c-ac23-dd18dc76c38d/manager/0.log" Jul 12 18:55:05.503207 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:05.503185 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-db78fd4c7-w2mwd_9506c9b2-3061-445c-ac23-dd18dc76c38d/copy-manifests/0.log" Jul 12 18:55:08.246318 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:08.246291 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bd558f979-vxgtx_07bec7be-8262-4463-94e7-a69a40481a95/kube-auth-proxy/0.log" Jul 12 18:55:08.264545 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:08.264517 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tbk4x_678cd7d5-7144-400c-a44f-c3b07b36fab9/istio-proxy/0.log" Jul 12 18:55:08.290953 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:08.290920 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tbk4x_678cd7d5-7144-400c-a44f-c3b07b36fab9/istio-proxy/1.log" Jul 12 18:55:08.317463 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:08.317428 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-9msj9_53951bfe-0dde-4ee0-9199-f2de0afb1c82/payload-pre-processing/0.log" Jul 12 18:55:08.352245 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:08.352213 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-v9qgm_629ea87f-db94-4c2c-ae5e-f1f48cb5d378/payload-processing/0.log" Jul 12 18:55:09.749184 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:09.749145 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bd558f979-vxgtx_07bec7be-8262-4463-94e7-a69a40481a95/kube-auth-proxy/0.log" Jul 12 18:55:09.766205 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:09.766165 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tbk4x_678cd7d5-7144-400c-a44f-c3b07b36fab9/istio-proxy/0.log" Jul 12 18:55:09.796794 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:09.796748 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tbk4x_678cd7d5-7144-400c-a44f-c3b07b36fab9/istio-proxy/1.log" Jul 12 18:55:09.829275 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:09.829243 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-9msj9_53951bfe-0dde-4ee0-9199-f2de0afb1c82/payload-pre-processing/0.log" Jul 12 18:55:09.872712 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:09.872685 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-v9qgm_629ea87f-db94-4c2c-ae5e-f1f48cb5d378/payload-processing/0.log" Jul 12 18:55:13.180245 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:13.180215 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-psnvh_fc40b8cb-8b6d-4358-b1ed-298429a488cd/kube-multus-additional-cni-plugins/0.log" Jul 12 18:55:13.197617 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:13.197593 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-psnvh_fc40b8cb-8b6d-4358-b1ed-298429a488cd/egress-router-binary-copy/0.log" Jul 12 18:55:13.214029 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:13.214008 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-psnvh_fc40b8cb-8b6d-4358-b1ed-298429a488cd/cni-plugins/0.log" Jul 12 18:55:13.231115 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:13.231082 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-psnvh_fc40b8cb-8b6d-4358-b1ed-298429a488cd/bond-cni-plugin/0.log" Jul 12 18:55:13.249004 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:13.248979 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-psnvh_fc40b8cb-8b6d-4358-b1ed-298429a488cd/routeoverride-cni/0.log" Jul 12 18:55:13.266387 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:13.266367 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-psnvh_fc40b8cb-8b6d-4358-b1ed-298429a488cd/whereabouts-cni-bincopy/0.log" Jul 12 18:55:13.285332 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:13.285305 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-psnvh_fc40b8cb-8b6d-4358-b1ed-298429a488cd/whereabouts-cni/0.log" Jul 12 18:55:13.345657 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:13.345628 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-plz8d_03b8837e-cff8-4cd8-8cf7-b32bda7762a7/kube-multus/0.log" Jul 12 18:55:13.395373 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:13.395342 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jw4j5_6a42ac1b-6e6a-4a60-8d10-42c2765b38ec/network-metrics-daemon/0.log" Jul 12 18:55:13.410030 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:13.410008 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jw4j5_6a42ac1b-6e6a-4a60-8d10-42c2765b38ec/kube-rbac-proxy/0.log" Jul 12 18:55:14.435440 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:14.435410 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pq4vs_c63cd484-5494-4519-9bf0-35a9d9c84ceb/ovn-controller/0.log" Jul 12 18:55:14.449210 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:14.449186 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pq4vs_c63cd484-5494-4519-9bf0-35a9d9c84ceb/ovn-acl-logging/0.log" Jul 12 18:55:14.468387 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:14.468363 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pq4vs_c63cd484-5494-4519-9bf0-35a9d9c84ceb/kube-rbac-proxy-node/0.log" Jul 12 18:55:14.523593 ip-10-0-132-5 kubenswrapper[2576]: I0712 18:55:14.523561 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pq4vs_c63cd484-5494-4519-9bf0-35a9d9c84ceb/kube-rbac-proxy-ovn-metrics/0.log"