Jul 23 00:07:58.746635 ip-10-0-134-124 systemd[1]: Starting Kubernetes Kubelet... Jul 23 00:07:59.071166 ip-10-0-134-124 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 00:07:59.071166 ip-10-0-134-124 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 00:07:59.071166 ip-10-0-134-124 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 00:07:59.071166 ip-10-0-134-124 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 00:07:59.071166 ip-10-0-134-124 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 00:07:59.073075 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.072961 2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 00:07:59.077362 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077346 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:07:59.077362 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077362 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077365 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077368 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077372 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077375 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077378 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077381 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077384 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077387 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077391 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077401 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077404 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077407 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077409 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077412 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077414 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077417 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077419 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077422 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:07:59.077428 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077425 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077427 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077430 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077432 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077435 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077438 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077440 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077443 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077446 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077450 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077454 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077458 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077461 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077463 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077468 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077471 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077473 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077476 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077479 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:07:59.077869 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077482 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077485 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077488 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077491 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077493 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077497 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077499 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077502 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077504 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077507 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077509 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077512 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077514 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077517 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077519 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077522 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077525 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077528 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:07:59.078319 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.077530 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078655 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078661 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078664 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078668 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078671 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078674 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078677 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078679 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078682 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078684 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078687 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078689 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078691 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078694 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078696 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078699 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078701 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078705 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078708 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:07:59.078800 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078713 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078715 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078718 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078720 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078723 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078725 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078728 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078731 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078733 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078736 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078739 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078741 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078744 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078746 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078749 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078752 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078755 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078758 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078760 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:07:59.079273 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078763 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078766 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.078768 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079493 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079499 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079502 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079505 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079508 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079511 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079514 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079516 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079519 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079522 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079524 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079529 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079532 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079535 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079538 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079541 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:07:59.079751 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079543 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079546 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079548 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079550 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079553 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079556 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079558 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079561 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079563 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079566 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079568 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079571 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079573 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079576 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079578 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079583 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079585 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079588 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079590 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079593 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:07:59.080557 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079595 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079598 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079601 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079603 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079606 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079608 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079611 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079613 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079616 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079618 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079621 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079623 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079625 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079628 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079630 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079633 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079636 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079638 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079641 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:07:59.081218 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079645 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079648 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079651 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079654 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079657 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079659 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079662 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079664 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079668 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079670 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079673 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079675 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079678 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079680 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079683 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079685 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079688 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079690 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079693 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079695 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:07:59.081702 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079698 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079700 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079703 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079705 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079708 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079792 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079797 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079800 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079803 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079806 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079809 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079812 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079815 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079817 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079820 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079825 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079829 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079836 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079839 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079842 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:07:59.082184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079845 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079849 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079852 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.079855 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081406 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081417 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081421 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081424 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081427 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081430 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081433 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081436 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081440 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081443 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081446 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081448 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081451 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081454 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081456 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:07:59.082686 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081459 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081462 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081465 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081468 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081471 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081474 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081476 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081479 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081482 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081485 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081487 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081490 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081492 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081495 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081497 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081500 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081503 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081505 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081509 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081511 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:07:59.083140 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081514 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081516 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081519 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081521 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081524 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081527 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081529 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081532 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081536 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081540 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081543 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081545 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081548 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081550 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081553 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081555 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081558 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081560 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081563 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081565 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:07:59.083675 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081568 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081570 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081573 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081576 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081578 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081581 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081583 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081586 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081590 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081594 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081598 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081602 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081606 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081609 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081612 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081615 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081617 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081620 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081623 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:07:59.084148 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081625 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081628 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081631 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081634 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081636 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081639 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081641 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081644 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081646 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081648 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081651 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081653 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081656 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081658 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081662 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081665 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081668 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081670 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081673 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081675 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:07:59.084616 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081678 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081680 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081690 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081693 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.081696 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081773 2555 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081784 2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081793 2555 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081800 2555 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081806 2555 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081811 2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081815 2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081820 2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081823 2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081826 2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081829 2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081832 2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081836 2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081839 2555 flags.go:64] FLAG: --cgroup-root="" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081841 2555 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081844 2555 flags.go:64] FLAG: --client-ca-file="" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081847 2555 flags.go:64] FLAG: --cloud-provider="external" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081850 2555 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 00:07:59.085092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081855 2555 flags.go:64] FLAG: --cluster-domain="" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081857 2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081860 2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081864 2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081867 2555 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081871 2555 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081874 2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081877 2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081880 2555 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081883 2555 flags.go:64] FLAG: --contention-profiling="false" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081886 2555 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081889 2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081899 2555 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081903 2555 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081907 2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081910 2555 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081913 2555 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081915 2555 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081918 2555 flags.go:64] FLAG: --enable-server="true" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081921 2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081931 2555 flags.go:64] FLAG: --event-burst="100" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081934 2555 flags.go:64] FLAG: --event-qps="50" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081937 2555 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081940 2555 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 00:07:59.085664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081943 2555 flags.go:64] FLAG: --eviction-hard="" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081947 2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081950 2555 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081953 2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081956 2555 flags.go:64] FLAG: --eviction-soft="" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081959 2555 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081961 2555 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081964 2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081967 2555 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081970 2555 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081972 2555 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081981 2555 flags.go:64] FLAG: --feature-gates="" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081985 2555 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081988 2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081991 2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081994 2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.081997 2555 flags.go:64] FLAG: --healthz-port="10248" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082000 2555 flags.go:64] FLAG: --help="false" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082003 2555 flags.go:64] FLAG: --hostname-override="ip-10-0-134-124.ec2.internal" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082006 2555 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082009 2555 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082018 2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082022 2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 00:07:59.086357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082025 2555 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082028 2555 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082031 2555 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082034 2555 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082037 2555 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082039 2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082042 2555 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082045 2555 flags.go:64] FLAG: --kube-reserved="" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082048 2555 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082050 2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082053 2555 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082056 2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082059 2555 flags.go:64] FLAG: --lock-file="" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082062 2555 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082065 2555 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082068 2555 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082073 2555 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082076 2555 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082078 2555 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082081 2555 flags.go:64] FLAG: --logging-format="text" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082085 2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082088 2555 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082091 2555 flags.go:64] FLAG: --manifest-url="" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082094 2555 flags.go:64] FLAG: --manifest-url-header="" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082099 2555 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 00:07:59.086943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082102 2555 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082106 2555 flags.go:64] FLAG: --max-pods="110" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082112 2555 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082115 2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082118 2555 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082120 2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082129 2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082132 2555 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082135 2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082149 2555 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082152 2555 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082155 2555 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082158 2555 flags.go:64] FLAG: --pod-cidr="" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082161 2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082166 2555 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082169 2555 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082172 2555 flags.go:64] FLAG: --pods-per-core="0" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082175 2555 flags.go:64] FLAG: --port="10250" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082178 2555 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082181 2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0456659179f6e31fe" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082184 2555 flags.go:64] FLAG: --qos-reserved="" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082187 2555 flags.go:64] FLAG: --read-only-port="10255" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082190 2555 flags.go:64] FLAG: --register-node="true" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082193 2555 flags.go:64] FLAG: --register-with-taints="" Jul 23 00:07:59.087622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082196 2555 flags.go:64] FLAG: --registry-burst="10" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082199 2555 flags.go:64] FLAG: --registry-qps="5" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082202 2555 flags.go:64] FLAG: --reserved-cpus="" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082206 2555 flags.go:64] FLAG: --reserved-memory="" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082210 2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082213 2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082216 2555 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082219 2555 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082221 2555 flags.go:64] FLAG: --runonce="false" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082224 2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082227 2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082232 2555 flags.go:64] FLAG: --seccomp-default="false" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082234 2555 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082237 2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082240 2555 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082249 2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082253 2555 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082255 2555 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082258 2555 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082261 2555 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082264 2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082267 2555 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082269 2555 flags.go:64] FLAG: --system-cgroups="" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082272 2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082277 2555 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 00:07:59.088244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082280 2555 flags.go:64] FLAG: --tls-cert-file="" Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082283 2555 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082292 2555 flags.go:64] FLAG: --tls-min-version="" Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082295 2555 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082297 2555 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082300 2555 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082303 2555 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082306 2555 flags.go:64] FLAG: --v="2" Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082310 2555 flags.go:64] FLAG: --version="false" Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082314 2555 flags.go:64] FLAG: --vmodule="" Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082318 2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082322 2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082462 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082466 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082469 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082471 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082474 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082478 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082483 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082487 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082491 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:07:59.088885 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082494 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082497 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082507 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082510 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082512 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082515 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082517 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082520 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082523 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082525 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082528 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082530 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082534 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082537 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082540 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082543 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082546 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082548 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082551 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082553 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:07:59.089404 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082556 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082560 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082563 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082566 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082568 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082571 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082574 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082576 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082579 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082583 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082585 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082588 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082591 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082593 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082596 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082604 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082606 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082609 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082611 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082614 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:07:59.089886 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082616 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082619 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082621 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082624 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082627 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082629 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082632 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082635 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082637 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082639 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082642 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082644 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082647 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082650 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082653 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082655 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082658 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082660 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082662 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082665 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:07:59.090403 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082667 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082671 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082674 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082676 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082679 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082681 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082684 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082686 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082695 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082698 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082700 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082703 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082705 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082708 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082710 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082713 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082716 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082718 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082721 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082723 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:07:59.090873 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082726 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:07:59.091369 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082728 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:07:59.091369 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082731 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:07:59.091369 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082733 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:07:59.091369 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082735 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:07:59.091369 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082740 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:07:59.091369 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082742 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:07:59.091369 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082745 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:07:59.091369 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082747 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:07:59.091369 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.082750 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:07:59.091369 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.082755 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 00:07:59.091369 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.089324 2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 00:07:59.091369 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.089352 2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 00:07:59.091369 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.089371 2555 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 00:07:59.091369 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.089377 2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 00:07:59.091369 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089430 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089435 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089439 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089442 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089445 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089448 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089451 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089454 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089457 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089459 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089462 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089465 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089468 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089470 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089473 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089476 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089480 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089484 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089487 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:07:59.091746 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089490 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089493 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089495 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089498 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089500 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089503 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089506 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089509 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089512 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089516 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089521 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089524 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089527 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089529 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089532 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089534 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089537 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089539 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089542 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089545 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:07:59.092203 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089547 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089550 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089553 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089556 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089558 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089561 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089563 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089566 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089569 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089571 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089574 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089576 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089579 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089581 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089584 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089586 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089589 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089591 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089595 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089597 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:07:59.092689 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089600 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089603 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089606 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089609 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089612 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089614 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089617 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089619 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089622 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089624 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089627 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089631 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089634 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089637 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089640 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089643 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089646 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089648 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089651 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:07:59.093212 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089654 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089656 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089659 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089661 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089664 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089666 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089669 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089671 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089673 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089676 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089678 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089681 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089684 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089686 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089689 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089692 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089695 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089699 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089701 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089704 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:07:59.093690 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089706 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:07:59.094184 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.089711 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 00:07:59.094184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089821 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:07:59.094184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089825 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:07:59.094184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089828 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:07:59.094184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089831 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:07:59.094184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089833 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:07:59.094184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089836 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:07:59.094184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089838 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:07:59.094184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089843 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:07:59.094184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089846 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:07:59.094184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089849 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:07:59.094184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089852 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:07:59.094184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089855 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:07:59.094184 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089857 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089860 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089862 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089865 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089868 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089870 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089873 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089876 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089878 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089881 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089883 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089886 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089889 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089892 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089895 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089898 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089901 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089903 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089906 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089909 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:07:59.094536 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089911 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089914 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089917 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089920 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089922 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089925 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089928 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089932 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089935 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089938 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089940 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089943 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089946 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089948 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089951 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089954 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089956 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089959 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089961 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:07:59.095023 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089964 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089967 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089969 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089972 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089974 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089977 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089979 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089982 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089984 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089987 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089990 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089993 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089995 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.089998 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090000 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090003 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090005 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090008 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090010 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090013 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:07:59.095500 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090015 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090018 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090020 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090023 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090025 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090028 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090030 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090033 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090035 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090038 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090040 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090043 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090045 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090048 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090051 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090053 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090056 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090058 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090061 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090064 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:07:59.096008 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090067 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:07:59.096549 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090070 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:07:59.096549 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090073 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:07:59.096549 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090076 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:07:59.096549 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090078 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:07:59.096549 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090081 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:07:59.096549 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090084 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:07:59.096549 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:07:59.090086 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:07:59.096549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.090091 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 00:07:59.096549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.090202 2555 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 00:07:59.096549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.095392 2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 00:07:59.096549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.096506 2555 server.go:1019] "Starting client certificate rotation" Jul 23 00:07:59.096917 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.096621 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 00:07:59.096917 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.096674 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 00:07:59.098603 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.098587 2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 00:07:59.099811 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.099791 2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 00:07:59.107410 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.107397 2555 log.go:25] "Validated CRI v1 runtime API" Jul 23 00:07:59.110085 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.110070 2555 log.go:25] "Validated CRI v1 image API" Jul 23 00:07:59.110821 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.110798 2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 00:07:59.112847 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.112830 2555 fs.go:135] Filesystem UUIDs: map[6f4330fc-5e7e-447c-8e6a-859e124842fc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b2856a8c-95e9-4e22-ad22-f18fc572394f:/dev/nvme0n1p3] Jul 23 00:07:59.112921 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.112846 2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 00:07:59.117258 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.117147 2555 manager.go:216] Machine: {Timestamp:2026-07-23 00:07:59.116007225 +0000 UTC m=+0.272807467 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098081 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e7d7d4cf31c46fd6226350889ca89 SystemUUID:ec2e7d7d-4cf3-1c46-fd62-26350889ca89 BootID:accfe57d-159f-4245-b831-ced8de9a06a4 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4a:f9:6e:10:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4a:f9:6e:10:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:41:79:57:71:c3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 00:07:59.117258 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.117246 2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 00:07:59.117389 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.117360 2555 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 00:07:59.120110 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.120085 2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 00:07:59.120261 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.120114 2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-124.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 00:07:59.120769 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.120758 2555 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 00:07:59.120821 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.120773 2555 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 00:07:59.120821 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.120791 2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 00:07:59.120821 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.120811 2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 00:07:59.122135 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.122123 2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 00:07:59.125851 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.125838 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 23 00:07:59.126001 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.125990 2555 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 00:07:59.126563 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.126553 2555 kubelet.go:486] "Attempting to sync node with API server" Jul 23 00:07:59.126626 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.126569 2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 00:07:59.126626 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.126589 2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 00:07:59.126626 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.126602 2555 kubelet.go:398] "Adding apiserver pod source" Jul 23 00:07:59.126626 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.126614 2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 00:07:59.126993 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.126970 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 00:07:59.126993 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.126993 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 00:07:59.128742 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.128725 2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 00:07:59.130393 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.130380 2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 00:07:59.130454 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.130410 2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 00:07:59.131434 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.131424 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 00:07:59.131473 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.131440 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 00:07:59.131473 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.131446 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 00:07:59.131473 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.131452 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 00:07:59.131473 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.131461 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 00:07:59.131473 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.131466 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 00:07:59.131473 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.131475 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 00:07:59.131473 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.131480 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 00:07:59.131662 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.131486 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 00:07:59.131662 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.131492 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 00:07:59.131662 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.131500 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 00:07:59.131662 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.131506 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 00:07:59.132227 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.132216 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 00:07:59.132261 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.132230 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 00:07:59.135231 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.135217 2555 server.go:1268] "Started kubelet" Jul 23 00:07:59.135339 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.135298 2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 00:07:59.135393 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.135310 2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 00:07:59.135439 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.135390 2555 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 00:07:59.136071 ip-10-0-134-124 systemd[1]: Started Kubernetes Kubelet. Jul 23 00:07:59.137047 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.137012 2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 00:07:59.137514 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.137494 2555 server.go:310] "Adding debug handlers to kubelet server" Jul 23 00:07:59.140356 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:07:59.140317 2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 00:07:59.143138 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.143121 2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 00:07:59.143918 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.143876 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 00:07:59.143992 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.143919 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 00:07:59.144319 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:07:59.144300 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found" Jul 23 00:07:59.144733 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.144718 2555 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 00:07:59.144842 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.144821 2555 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 00:07:59.145029 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.144982 2555 reconstruct.go:97] "Volume reconstruction finished" Jul 23 00:07:59.145029 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.144994 2555 reconciler.go:29] "Reconciler: start to sync state" Jul 23 00:07:59.145233 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:07:59.145190 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 00:07:59.145274 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.145231 2555 factory.go:55] Registering systemd factory Jul 23 00:07:59.145274 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.145251 2555 factory.go:223] Registration of the systemd container factory successfully Jul 23 00:07:59.145391 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.145305 2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 00:07:59.145391 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.145385 2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 00:07:59.145543 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.145524 2555 factory.go:153] Registering CRI-O factory Jul 23 00:07:59.145543 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.145545 2555 factory.go:223] Registration of the crio container factory successfully Jul 23 00:07:59.145721 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:07:59.145610 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 00:07:59.145721 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.145654 2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 00:07:59.145721 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.145680 2555 factory.go:103] Registering Raw factory Jul 23 00:07:59.145827 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.145744 2555 manager.go:1196] Started watching for new ooms in manager Jul 23 00:07:59.146485 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.146472 2555 manager.go:349] Starting recovery of all containers Jul 23 00:07:59.148345 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:07:59.144742 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-124.ec2.internal.18c4c290b64dcd91 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-124.ec2.internal,UID:ip-10-0-134-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-124.ec2.internal,},FirstTimestamp:2026-07-23 00:07:59.135190417 +0000 UTC m=+0.291990658,LastTimestamp:2026-07-23 00:07:59.135190417 +0000 UTC m=+0.291990658,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-124.ec2.internal,}" Jul 23 00:07:59.148625 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:07:59.148596 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 00:07:59.148625 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:07:59.148612 2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 00:07:59.158757 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.158742 2555 manager.go:354] Recovery completed Jul 23 00:07:59.162850 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.162837 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:07:59.164387 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.164364 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:07:59.164458 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.164398 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:07:59.164458 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.164408 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:07:59.164925 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.164913 2555 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 00:07:59.164925 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.164924 2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 00:07:59.164998 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.164941 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 23 00:07:59.164998 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.164977 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:07:59.165718 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.165705 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:07:59.165815 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.165731 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:07:59.165815 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.165740 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:07:59.167121 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.167110 2555 policy_none.go:49] "None policy: Start" Jul 23 00:07:59.167155 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.167125 2555 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 00:07:59.167155 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.167135 2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 00:07:59.169014 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.169005 2555 policy_none.go:47] "Start" Jul 23 00:07:59.171340 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:07:59.171253 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-124.ec2.internal.18c4c290b80b4b46 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-124.ec2.internal,UID:ip-10-0-134-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-124.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-124.ec2.internal,},FirstTimestamp:2026-07-23 00:07:59.164386118 +0000 UTC m=+0.321186359,LastTimestamp:2026-07-23 00:07:59.164386118 +0000 UTC m=+0.321186359,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-124.ec2.internal,}" Jul 23 00:07:59.185002 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:07:59.184823 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-124.ec2.internal.18c4c290b80b8ccf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-124.ec2.internal,UID:ip-10-0-134-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-124.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-124.ec2.internal,},FirstTimestamp:2026-07-23 00:07:59.164402895 +0000 UTC m=+0.321203136,LastTimestamp:2026-07-23 00:07:59.164402895 +0000 UTC m=+0.321203136,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-124.ec2.internal,}" Jul 23 00:07:59.196103 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:07:59.196045 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-124.ec2.internal.18c4c290b80bb269 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-124.ec2.internal,UID:ip-10-0-134-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-124.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-124.ec2.internal,},FirstTimestamp:2026-07-23 00:07:59.164412521 +0000 UTC m=+0.321212762,LastTimestamp:2026-07-23 00:07:59.164412521 +0000 UTC m=+0.321212762,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-124.ec2.internal,}" Jul 23 00:07:59.203212 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.203196 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5gv8k" Jul 23 00:07:59.206889 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.206874 2555 manager.go:337] "Starting Device Plugin manager" Jul 23 00:07:59.206964 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:07:59.206904 2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 00:07:59.206964 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.206914 2555 server.go:87] "Starting device plugin registration server" Jul 23 00:07:59.207137 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.207122 2555 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 00:07:59.207228 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.207139 2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 00:07:59.207228 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.207200 2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 00:07:59.207381 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.207275 2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 00:07:59.207381 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.207283 2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 00:07:59.207880 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:07:59.207783 2555 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-124.ec2.internal.18c4c290b80b4b46\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-124.ec2.internal.18c4c290b80b4b46 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-124.ec2.internal,UID:ip-10-0-134-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-124.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-124.ec2.internal,},FirstTimestamp:2026-07-23 00:07:59.164386118 +0000 UTC m=+0.321186359,LastTimestamp:2026-07-23 00:07:59.165716936 +0000 UTC m=+0.322517176,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-124.ec2.internal,}" Jul 23 00:07:59.207970 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:07:59.207895 2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 00:07:59.207970 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:07:59.207943 2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-124.ec2.internal\" not found" Jul 23 00:07:59.212715 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:07:59.212697 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5gv8k" Jul 23 00:07:59.217633 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:07:59.217572 2555 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-124.ec2.internal.18c4c290b80b8ccf\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-124.ec2.internal.18c4c290b80b8ccf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-124.ec2.internal,UID:ip-10-0-134-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-124.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-124.ec2.internal,},FirstTimestamp:2026-07-23 00:07:59.164402895 +0000 UTC m=+0.321203136,LastTimestamp:2026-07-23 00:07:59.165735162 +0000 UTC m=+0.322535403,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-124.ec2.internal,}" Jul 23 00:08:00.494740 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.494700 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 00:08:00.495965 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.495946 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 00:08:00.496018 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.496000 2555 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 00:08:00.496061 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.496028 2555 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 00:08:00.496109 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:00.496071 2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 00:08:00.498404 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.498364 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:08:00.504256 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.504236 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:08:00.504989 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.504971 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:08:00.505069 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.505006 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:08:00.505069 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.505018 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:08:00.505069 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.505041 2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-124.ec2.internal" Jul 23 00:08:00.512962 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.512945 2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-124.ec2.internal" Jul 23 00:08:00.513051 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:00.512972 2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-124.ec2.internal\": node \"ip-10-0-134-124.ec2.internal\" not found" Jul 23 00:08:00.527146 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:00.527085 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found" Jul 23 00:08:00.596832 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.596800 2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal"] Jul 23 00:08:00.596976 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.596887 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:08:00.598414 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.598376 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:08:00.598514 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.598433 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:08:00.598514 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.598448 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:08:00.600855 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.600843 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:08:00.600967 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.600954 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal" Jul 23 00:08:00.601017 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.600983 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:08:00.601617 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.601604 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:08:00.601617 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.601614 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:08:00.601711 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.601628 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:08:00.601711 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.601640 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:08:00.601711 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.601633 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:08:00.601711 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.601661 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:08:00.603849 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.603832 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal" Jul 23 00:08:00.603927 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.603856 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:08:00.604709 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.604692 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:08:00.604816 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.604722 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:08:00.604816 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.604736 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:08:00.624943 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:00.624919 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-124.ec2.internal\" not found" node="ip-10-0-134-124.ec2.internal" Jul 23 00:08:00.627144 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:00.627129 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found" Jul 23 00:08:00.628313 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:00.628296 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-124.ec2.internal\" not found" node="ip-10-0-134-124.ec2.internal" Jul 23 00:08:00.727788 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:00.727759 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found" Jul 23 00:08:00.743382 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.743360 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/524c78bd725b7ef91dc2abac8d4936d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal\" (UID: \"524c78bd725b7ef91dc2abac8d4936d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal" Jul 23 00:08:00.743460 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.743410 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/524c78bd725b7ef91dc2abac8d4936d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal\" (UID: \"524c78bd725b7ef91dc2abac8d4936d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal" Jul 23 00:08:00.743460 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.743429 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c386b32f5b824bf53480b029cd79f9a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-124.ec2.internal\" (UID: \"6c386b32f5b824bf53480b029cd79f9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal" Jul 23 00:08:00.828933 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:00.828856 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found" Jul 23 00:08:00.844285 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.844263 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/524c78bd725b7ef91dc2abac8d4936d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal\" (UID: \"524c78bd725b7ef91dc2abac8d4936d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal" Jul 23 00:08:00.844334 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.844290 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/524c78bd725b7ef91dc2abac8d4936d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal\" (UID: \"524c78bd725b7ef91dc2abac8d4936d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal" Jul 23 00:08:00.844334 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.844313 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c386b32f5b824bf53480b029cd79f9a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-124.ec2.internal\" (UID: \"6c386b32f5b824bf53480b029cd79f9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal" Jul 23 00:08:00.844421 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.844348 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/524c78bd725b7ef91dc2abac8d4936d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal\" (UID: \"524c78bd725b7ef91dc2abac8d4936d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal" Jul 23 00:08:00.844421 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.844354 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/524c78bd725b7ef91dc2abac8d4936d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal\" (UID: \"524c78bd725b7ef91dc2abac8d4936d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal" Jul 23 00:08:00.844421 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.844348 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c386b32f5b824bf53480b029cd79f9a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-124.ec2.internal\" (UID: \"6c386b32f5b824bf53480b029cd79f9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal" Jul 23 00:08:00.928507 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.928461 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal" Jul 23 00:08:00.929547 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:00.929532 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found" Jul 23 00:08:00.931700 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:00.931682 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal" Jul 23 00:08:01.030494 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:01.030461 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found" Jul 23 00:08:01.131080 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:01.130995 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found" Jul 23 00:08:01.201725 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:01.201701 2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:08:01.231468 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:01.231430 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found" Jul 23 00:08:01.292068 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:01.292043 2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 00:08:01.292507 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:01.292183 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 00:08:01.292507 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:01.292223 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 00:08:01.332599 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:01.332550 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found" Jul 23 00:08:01.413760 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:01.413669 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 00:03:00 +0000 UTC" deadline="2028-03-23 11:52:35.126807684 +0000 UTC" Jul 23 00:08:01.413760 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:01.413717 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14627h44m33.713107073s" Jul 23 00:08:01.432853 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:01.432817 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found" Jul 23 00:08:01.533725 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:01.533695 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-124.ec2.internal\" not found" Jul 23 00:08:01.557198 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:01.557177 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:08:01.558430 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:01.558408 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:08:01.558544 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:01.558443 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:08:01.558544 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:01.558458 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:08:01.597421 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:01.597383 2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:08:01.598638 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:08:01.598613 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524c78bd725b7ef91dc2abac8d4936d9.slice/crio-65110aead64ab695799869602fa63489944966ee840ced948f29d692cb58fc66 WatchSource:0}: Error finding container 65110aead64ab695799869602fa63489944966ee840ced948f29d692cb58fc66: Status 404 returned error can't find the container with id 65110aead64ab695799869602fa63489944966ee840ced948f29d692cb58fc66 Jul 23 00:08:01.599129 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:08:01.599113 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c386b32f5b824bf53480b029cd79f9a.slice/crio-41a281829c1ad786ca640d1569ead1ff95321180f413da591cf9aa0b8fc2d9e3 WatchSource:0}: Error finding container 41a281829c1ad786ca640d1569ead1ff95321180f413da591cf9aa0b8fc2d9e3: Status 404 returned error can't find the container with id 41a281829c1ad786ca640d1569ead1ff95321180f413da591cf9aa0b8fc2d9e3 Jul 23 00:08:01.603455 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:01.603440 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:08:01.640673 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:01.640645 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal" Jul 23 00:08:01.647782 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:01.647760 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 00:08:01.649350 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:01.649337 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal" Jul 23 00:08:01.657905 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:01.657883 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 00:08:01.882060 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:01.881979 2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:08:02.196257 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.196168 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:08:02.323867 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.323626 2555 apiserver.go:52] "Watching apiserver" Jul 23 00:08:02.330443 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.330344 2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 00:08:02.331698 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.331661 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-jbwvd","kube-system/konnectivity-agent-2x8c9","kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4","openshift-cluster-node-tuning-operator/tuned-ssldm","openshift-image-registry/node-ca-pn5jp","openshift-multus/multus-b699m","openshift-dns/node-resolver-t75zr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal","openshift-multus/multus-additional-cni-plugins-m9njl","openshift-multus/network-metrics-daemon-czq2t","openshift-network-diagnostics/network-check-target-zgb4z","openshift-network-operator/iptables-alerter-6nxnc","openshift-ovn-kubernetes/ovnkube-node-d8rnh"] Jul 23 00:08:02.340199 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.340167 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b699m" Jul 23 00:08:02.340799 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.340773 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 00:08:02.342348 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.342330 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6nxnc" Jul 23 00:08:02.343572 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.343550 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 00:08:02.343664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.343648 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-txrt7\"" Jul 23 00:08:02.343906 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.343880 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 00:08:02.344030 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.344008 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 00:08:02.344201 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.344136 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 00:08:02.345207 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.344806 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.345207 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.344870 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:08:02.345631 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.345611 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kw644\"" Jul 23 00:08:02.345707 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.345662 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 00:08:02.346305 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.346285 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 00:08:02.347316 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.347296 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.348890 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.348871 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 00:08:02.349165 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.349145 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 00:08:02.349516 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.349500 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 00:08:02.349712 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.349695 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 00:08:02.349777 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.349760 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.350232 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.350215 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 00:08:02.350450 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.350433 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 00:08:02.350632 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.350545 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cbgfl\"" Jul 23 00:08:02.351471 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.351450 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 00:08:02.351758 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.351739 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 00:08:02.352250 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.352211 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h9gf4\"" Jul 23 00:08:02.352547 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.352525 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pn5jp" Jul 23 00:08:02.352897 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.352873 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-cnibin\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.352964 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.352934 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-var-lib-cni-bin\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.353016 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.352966 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54398313-28cf-4172-a8d3-dcc8ea7ca95b-host-slash\") pod \"iptables-alerter-6nxnc\" (UID: \"54398313-28cf-4172-a8d3-dcc8ea7ca95b\") " pod="openshift-network-operator/iptables-alerter-6nxnc" Jul 23 00:08:02.353016 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353001 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-os-release\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.353116 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353026 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-multus-socket-dir-parent\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.353116 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353051 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-var-lib-cni-multus\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.353116 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353074 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-hostroot\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.353116 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353097 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60bc3129-fc54-4d47-a7b4-834eb559fd37-multus-daemon-config\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.353296 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353118 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-run-multus-certs\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.353296 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353141 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-system-cni-dir\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.353296 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353165 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-run-k8s-cni-cncf-io\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.353296 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353188 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-run-netns\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.353296 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353217 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-var-lib-kubelet\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.353296 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353240 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-multus-conf-dir\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.353296 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353261 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-etc-kubernetes\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.353632 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353297 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prwrt\" (UniqueName: \"kubernetes.io/projected/60bc3129-fc54-4d47-a7b4-834eb559fd37-kube-api-access-prwrt\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.353632 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353321 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54398313-28cf-4172-a8d3-dcc8ea7ca95b-iptables-alerter-script\") pod \"iptables-alerter-6nxnc\" (UID: \"54398313-28cf-4172-a8d3-dcc8ea7ca95b\") " pod="openshift-network-operator/iptables-alerter-6nxnc" Jul 23 00:08:02.353632 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353345 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-multus-cni-dir\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.353632 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353366 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60bc3129-fc54-4d47-a7b4-834eb559fd37-cni-binary-copy\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.353632 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353423 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfnqv\" (UniqueName: \"kubernetes.io/projected/54398313-28cf-4172-a8d3-dcc8ea7ca95b-kube-api-access-kfnqv\") pod \"iptables-alerter-6nxnc\" (UID: \"54398313-28cf-4172-a8d3-dcc8ea7ca95b\") " pod="openshift-network-operator/iptables-alerter-6nxnc" Jul 23 00:08:02.353632 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353531 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 00:08:02.353632 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353577 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 00:08:02.354000 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353534 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:08:02.354000 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353820 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zmzbv\"" Jul 23 00:08:02.354000 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.353965 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 00:08:02.354235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.354219 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t75zr" Jul 23 00:08:02.355120 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.355082 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 00:08:02.355211 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.355156 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 00:08:02.355343 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.355311 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 00:08:02.355755 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.355738 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mpw27\"" Jul 23 00:08:02.356716 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.356698 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:02.356808 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:02.356772 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:02.357201 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.357100 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 00:08:02.357555 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.357535 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ltqsf\"" Jul 23 00:08:02.357636 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.357584 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 00:08:02.359047 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.358976 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2x8c9" Jul 23 00:08:02.361317 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.361298 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.361719 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.361697 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x5smg\"" Jul 23 00:08:02.362035 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.362016 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 00:08:02.362110 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.362059 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 00:08:02.364132 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.364112 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:02.364222 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:02.364191 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:02.365757 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.365736 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 00:08:02.365942 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.365924 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bccq7\"" Jul 23 00:08:02.366357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.366339 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 00:08:02.366869 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.366851 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:02.366937 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:02.366923 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:02.373036 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.373013 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hkgz9" Jul 23 00:08:02.381457 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.381435 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hkgz9" Jul 23 00:08:02.443468 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.443441 2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 00:08:02.453961 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.453885 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57ss4\" (UniqueName: \"kubernetes.io/projected/f8507188-7090-4b88-940f-cf6f13963aea-kube-api-access-57ss4\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.453961 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.453927 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-run\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.453961 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.453955 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f8507188-7090-4b88-940f-cf6f13963aea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.454235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.453978 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-etc-openvswitch\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.454235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454002 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-etc-selinux\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.454235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454049 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-sys\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.454235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454073 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-lib-modules\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.454235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454110 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-sys-fs\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.454235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454138 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-systemd\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.454235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454163 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-run-netns\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.454235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454189 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-device-dir\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.454235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454221 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-etc-kubernetes\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.454619 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454241 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prwrt\" (UniqueName: \"kubernetes.io/projected/60bc3129-fc54-4d47-a7b4-834eb559fd37-kube-api-access-prwrt\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.454619 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454266 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8507188-7090-4b88-940f-cf6f13963aea-cni-binary-copy\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.454619 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454291 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znx59\" (UniqueName: \"kubernetes.io/projected/754c2d81-3f3e-44d4-8589-fec36d904b15-kube-api-access-znx59\") pod \"node-ca-pn5jp\" (UID: \"754c2d81-3f3e-44d4-8589-fec36d904b15\") " pod="openshift-image-registry/node-ca-pn5jp" Jul 23 00:08:02.454619 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454314 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-log-socket\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.454619 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454321 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-etc-kubernetes\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.454619 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454336 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-cni-netd\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.454619 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454365 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-kubernetes\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.454619 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454480 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-run-netns\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.454619 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454508 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-var-lib-kubelet\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.454619 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454583 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-run-netns\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.455050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454625 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76c4a8e8-f0b1-4468-996d-dae6883fdca4-agent-certs\") pod \"konnectivity-agent-2x8c9\" (UID: \"76c4a8e8-f0b1-4468-996d-dae6883fdca4\") " pod="kube-system/konnectivity-agent-2x8c9" Jul 23 00:08:02.455050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454646 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-var-lib-kubelet\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.455050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454651 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-host\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.455050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454690 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-slash\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.455050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454715 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3626e0e2-750a-4540-af36-dba6f4802240-ovnkube-config\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.455050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454739 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3626e0e2-750a-4540-af36-dba6f4802240-env-overrides\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.455050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454776 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3626e0e2-750a-4540-af36-dba6f4802240-ovnkube-script-lib\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.455050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454803 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54398313-28cf-4172-a8d3-dcc8ea7ca95b-host-slash\") pod \"iptables-alerter-6nxnc\" (UID: \"54398313-28cf-4172-a8d3-dcc8ea7ca95b\") " pod="openshift-network-operator/iptables-alerter-6nxnc" Jul 23 00:08:02.455050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454827 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/225d2d8c-3a75-47f8-9943-2f67e3c62e8b-tmp-dir\") pod \"node-resolver-t75zr\" (UID: \"225d2d8c-3a75-47f8-9943-2f67e3c62e8b\") " pod="openshift-dns/node-resolver-t75zr" Jul 23 00:08:02.455050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454851 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-run-ovn\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.455050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454876 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54398313-28cf-4172-a8d3-dcc8ea7ca95b-host-slash\") pod \"iptables-alerter-6nxnc\" (UID: \"54398313-28cf-4172-a8d3-dcc8ea7ca95b\") " pod="openshift-network-operator/iptables-alerter-6nxnc" Jul 23 00:08:02.455050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454875 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-registration-dir\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.455050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454921 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-var-lib-cni-multus\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.455050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454943 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-hostroot\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.455050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454966 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54398313-28cf-4172-a8d3-dcc8ea7ca95b-iptables-alerter-script\") pod \"iptables-alerter-6nxnc\" (UID: \"54398313-28cf-4172-a8d3-dcc8ea7ca95b\") " pod="openshift-network-operator/iptables-alerter-6nxnc" Jul 23 00:08:02.455050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.454989 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8507188-7090-4b88-940f-cf6f13963aea-cnibin\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.455050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455023 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f8507188-7090-4b88-940f-cf6f13963aea-os-release\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.455727 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455057 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8507188-7090-4b88-940f-cf6f13963aea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.455727 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455068 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-var-lib-cni-multus\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.455727 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455080 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs\") pod \"network-metrics-daemon-czq2t\" (UID: \"907325cd-64b7-4241-8969-a32a19bab628\") " pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:02.455727 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455103 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-systemd-units\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.455727 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455123 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/754c2d81-3f3e-44d4-8589-fec36d904b15-serviceca\") pod \"node-ca-pn5jp\" (UID: \"754c2d81-3f3e-44d4-8589-fec36d904b15\") " pod="openshift-image-registry/node-ca-pn5jp" Jul 23 00:08:02.455727 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455148 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-run-k8s-cni-cncf-io\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.455727 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455170 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-var-lib-kubelet\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.455727 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455190 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-tuned\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.455727 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455211 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkr2s\" (UniqueName: \"kubernetes.io/projected/06fa9895-13a2-45a5-bc6a-b618b99c5de4-kube-api-access-mkr2s\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.455727 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455232 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/225d2d8c-3a75-47f8-9943-2f67e3c62e8b-hosts-file\") pod \"node-resolver-t75zr\" (UID: \"225d2d8c-3a75-47f8-9943-2f67e3c62e8b\") " pod="openshift-dns/node-resolver-t75zr" Jul 23 00:08:02.455727 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455256 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8507188-7090-4b88-940f-cf6f13963aea-system-cni-dir\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.455727 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455279 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76c4a8e8-f0b1-4468-996d-dae6883fdca4-konnectivity-ca\") pod \"konnectivity-agent-2x8c9\" (UID: \"76c4a8e8-f0b1-4468-996d-dae6883fdca4\") " pod="kube-system/konnectivity-agent-2x8c9" Jul 23 00:08:02.455727 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455301 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret\") pod \"global-pull-secret-syncer-jbwvd\" (UID: \"5981d2c3-3af8-4f97-b00f-eed942743c86\") " pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:02.455727 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455332 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqsfw\" (UniqueName: \"kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw\") pod \"network-check-target-zgb4z\" (UID: \"00044a26-c2cd-42d4-98df-5b62a5084d8b\") " pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:02.455727 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455379 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-run-systemd\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.455727 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455424 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-node-log\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.456433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455449 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-cni-bin\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.456433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455494 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3626e0e2-750a-4540-af36-dba6f4802240-ovn-node-metrics-cert\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.456433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455520 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv7j6\" (UniqueName: \"kubernetes.io/projected/4546908a-9199-44a1-aae5-603ac8e91370-kube-api-access-hv7j6\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.456433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455591 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-cnibin\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.456433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455617 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-var-lib-cni-bin\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.456433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455648 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54398313-28cf-4172-a8d3-dcc8ea7ca95b-iptables-alerter-script\") pod \"iptables-alerter-6nxnc\" (UID: \"54398313-28cf-4172-a8d3-dcc8ea7ca95b\") " pod="openshift-network-operator/iptables-alerter-6nxnc" Jul 23 00:08:02.456433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455663 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06fa9895-13a2-45a5-bc6a-b618b99c5de4-tmp\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.456433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455691 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5981d2c3-3af8-4f97-b00f-eed942743c86-dbus\") pod \"global-pull-secret-syncer-jbwvd\" (UID: \"5981d2c3-3af8-4f97-b00f-eed942743c86\") " pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:02.456433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455026 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-hostroot\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.456433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455712 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/754c2d81-3f3e-44d4-8589-fec36d904b15-host\") pod \"node-ca-pn5jp\" (UID: \"754c2d81-3f3e-44d4-8589-fec36d904b15\") " pod="openshift-image-registry/node-ca-pn5jp" Jul 23 00:08:02.456433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455737 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-run-openvswitch\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.456433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455760 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-run-k8s-cni-cncf-io\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.456433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455761 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.456433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455801 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.456433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455826 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-sysctl-conf\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.456433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455895 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-os-release\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.456433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455924 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-cnibin\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.457189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455960 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-os-release\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.457189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455962 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-var-lib-cni-bin\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.457189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.455999 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-multus-socket-dir-parent\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.457189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456030 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-run-multus-certs\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.457189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456056 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-kubelet\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.457189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456084 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.457189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456075 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-host-run-multus-certs\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.457189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456111 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfnqv\" (UniqueName: \"kubernetes.io/projected/54398313-28cf-4172-a8d3-dcc8ea7ca95b-kube-api-access-kfnqv\") pod \"iptables-alerter-6nxnc\" (UID: \"54398313-28cf-4172-a8d3-dcc8ea7ca95b\") " pod="openshift-network-operator/iptables-alerter-6nxnc" Jul 23 00:08:02.457189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456136 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-system-cni-dir\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.457189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456160 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-multus-conf-dir\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.457189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456170 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-multus-socket-dir-parent\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.457189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456182 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-multus-cni-dir\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.457189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456207 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60bc3129-fc54-4d47-a7b4-834eb559fd37-cni-binary-copy\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.457189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456231 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-modprobe-d\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.457189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456255 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-sysconfig\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.457189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456279 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5qfq\" (UniqueName: \"kubernetes.io/projected/225d2d8c-3a75-47f8-9943-2f67e3c62e8b-kube-api-access-j5qfq\") pod \"node-resolver-t75zr\" (UID: \"225d2d8c-3a75-47f8-9943-2f67e3c62e8b\") " pod="openshift-dns/node-resolver-t75zr" Jul 23 00:08:02.457189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456302 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5981d2c3-3af8-4f97-b00f-eed942743c86-kubelet-config\") pod \"global-pull-secret-syncer-jbwvd\" (UID: \"5981d2c3-3af8-4f97-b00f-eed942743c86\") " pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:02.457820 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456325 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f8507188-7090-4b88-940f-cf6f13963aea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.457820 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456352 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mql7f\" (UniqueName: \"kubernetes.io/projected/907325cd-64b7-4241-8969-a32a19bab628-kube-api-access-mql7f\") pod \"network-metrics-daemon-czq2t\" (UID: \"907325cd-64b7-4241-8969-a32a19bab628\") " pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:02.457820 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456377 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-sysctl-d\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.457820 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456401 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-multus-conf-dir\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.457820 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456419 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-var-lib-openvswitch\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.457820 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456355 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-multus-cni-dir\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.457820 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456231 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60bc3129-fc54-4d47-a7b4-834eb559fd37-system-cni-dir\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.457820 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456460 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbd6z\" (UniqueName: \"kubernetes.io/projected/3626e0e2-750a-4540-af36-dba6f4802240-kube-api-access-fbd6z\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.457820 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456501 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-socket-dir\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.457820 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456545 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60bc3129-fc54-4d47-a7b4-834eb559fd37-multus-daemon-config\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.457820 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.456925 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60bc3129-fc54-4d47-a7b4-834eb559fd37-cni-binary-copy\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.457820 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.457063 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60bc3129-fc54-4d47-a7b4-834eb559fd37-multus-daemon-config\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.462027 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.462002 2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 00:08:02.466058 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.465813 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prwrt\" (UniqueName: \"kubernetes.io/projected/60bc3129-fc54-4d47-a7b4-834eb559fd37-kube-api-access-prwrt\") pod \"multus-b699m\" (UID: \"60bc3129-fc54-4d47-a7b4-834eb559fd37\") " pod="openshift-multus/multus-b699m" Jul 23 00:08:02.466218 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.466179 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfnqv\" (UniqueName: \"kubernetes.io/projected/54398313-28cf-4172-a8d3-dcc8ea7ca95b-kube-api-access-kfnqv\") pod \"iptables-alerter-6nxnc\" (UID: \"54398313-28cf-4172-a8d3-dcc8ea7ca95b\") " pod="openshift-network-operator/iptables-alerter-6nxnc" Jul 23 00:08:02.501078 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.501028 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal" event={"ID":"6c386b32f5b824bf53480b029cd79f9a","Type":"ContainerStarted","Data":"41a281829c1ad786ca640d1569ead1ff95321180f413da591cf9aa0b8fc2d9e3"} Jul 23 00:08:02.502209 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.502178 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal" event={"ID":"524c78bd725b7ef91dc2abac8d4936d9","Type":"ContainerStarted","Data":"65110aead64ab695799869602fa63489944966ee840ced948f29d692cb58fc66"} Jul 23 00:08:02.556938 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.556906 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-systemd\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.556938 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.556947 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-run-netns\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.557167 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.556973 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-device-dir\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.557167 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.556999 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8507188-7090-4b88-940f-cf6f13963aea-cni-binary-copy\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.557167 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557016 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-run-netns\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.557167 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557033 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-systemd\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.557167 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557023 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znx59\" (UniqueName: \"kubernetes.io/projected/754c2d81-3f3e-44d4-8589-fec36d904b15-kube-api-access-znx59\") pod \"node-ca-pn5jp\" (UID: \"754c2d81-3f3e-44d4-8589-fec36d904b15\") " pod="openshift-image-registry/node-ca-pn5jp" Jul 23 00:08:02.557167 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557081 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-log-socket\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.557167 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557086 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-device-dir\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.557167 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557108 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-cni-netd\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.557167 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557129 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-log-socket\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.557167 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557141 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-kubernetes\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.557167 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557163 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-cni-netd\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.557167 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557169 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76c4a8e8-f0b1-4468-996d-dae6883fdca4-agent-certs\") pod \"konnectivity-agent-2x8c9\" (UID: \"76c4a8e8-f0b1-4468-996d-dae6883fdca4\") " pod="kube-system/konnectivity-agent-2x8c9" Jul 23 00:08:02.557831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557192 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-host\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.557831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557206 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-kubernetes\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.557831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557214 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-slash\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.557831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557239 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3626e0e2-750a-4540-af36-dba6f4802240-ovnkube-config\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.557831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557285 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-host\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.557831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557315 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3626e0e2-750a-4540-af36-dba6f4802240-env-overrides\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.557831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557338 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3626e0e2-750a-4540-af36-dba6f4802240-ovnkube-script-lib\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.557831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557363 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/225d2d8c-3a75-47f8-9943-2f67e3c62e8b-tmp-dir\") pod \"node-resolver-t75zr\" (UID: \"225d2d8c-3a75-47f8-9943-2f67e3c62e8b\") " pod="openshift-dns/node-resolver-t75zr" Jul 23 00:08:02.557831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557385 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-run-ovn\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.557831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557442 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-registration-dir\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.557831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557468 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8507188-7090-4b88-940f-cf6f13963aea-cnibin\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.557831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557502 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f8507188-7090-4b88-940f-cf6f13963aea-os-release\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.557831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557528 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8507188-7090-4b88-940f-cf6f13963aea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.557831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557551 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs\") pod \"network-metrics-daemon-czq2t\" (UID: \"907325cd-64b7-4241-8969-a32a19bab628\") " pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:02.557831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557574 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-systemd-units\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.557831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557596 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/754c2d81-3f3e-44d4-8589-fec36d904b15-serviceca\") pod \"node-ca-pn5jp\" (UID: \"754c2d81-3f3e-44d4-8589-fec36d904b15\") " pod="openshift-image-registry/node-ca-pn5jp" Jul 23 00:08:02.557831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557618 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-var-lib-kubelet\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.558679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557652 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-tuned\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.558679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557677 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkr2s\" (UniqueName: \"kubernetes.io/projected/06fa9895-13a2-45a5-bc6a-b618b99c5de4-kube-api-access-mkr2s\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.558679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557700 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/225d2d8c-3a75-47f8-9943-2f67e3c62e8b-hosts-file\") pod \"node-resolver-t75zr\" (UID: \"225d2d8c-3a75-47f8-9943-2f67e3c62e8b\") " pod="openshift-dns/node-resolver-t75zr" Jul 23 00:08:02.558679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557725 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8507188-7090-4b88-940f-cf6f13963aea-system-cni-dir\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.558679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557751 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76c4a8e8-f0b1-4468-996d-dae6883fdca4-konnectivity-ca\") pod \"konnectivity-agent-2x8c9\" (UID: \"76c4a8e8-f0b1-4468-996d-dae6883fdca4\") " pod="kube-system/konnectivity-agent-2x8c9" Jul 23 00:08:02.558679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557772 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret\") pod \"global-pull-secret-syncer-jbwvd\" (UID: \"5981d2c3-3af8-4f97-b00f-eed942743c86\") " pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:02.558679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557796 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqsfw\" (UniqueName: \"kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw\") pod \"network-check-target-zgb4z\" (UID: \"00044a26-c2cd-42d4-98df-5b62a5084d8b\") " pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:02.558679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557820 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-run-systemd\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.558679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557845 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-node-log\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.558679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557865 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8507188-7090-4b88-940f-cf6f13963aea-cni-binary-copy\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.558679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557869 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3626e0e2-750a-4540-af36-dba6f4802240-ovnkube-config\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.558679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557870 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-cni-bin\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.558679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557921 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3626e0e2-750a-4540-af36-dba6f4802240-ovn-node-metrics-cert\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.558679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557939 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-slash\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.558679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.557947 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv7j6\" (UniqueName: \"kubernetes.io/projected/4546908a-9199-44a1-aae5-603ac8e91370-kube-api-access-hv7j6\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.558679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558283 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-var-lib-kubelet\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.558679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558297 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/754c2d81-3f3e-44d4-8589-fec36d904b15-serviceca\") pod \"node-ca-pn5jp\" (UID: \"754c2d81-3f3e-44d4-8589-fec36d904b15\") " pod="openshift-image-registry/node-ca-pn5jp" Jul 23 00:08:02.559549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558336 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06fa9895-13a2-45a5-bc6a-b618b99c5de4-tmp\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.559549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558361 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5981d2c3-3af8-4f97-b00f-eed942743c86-dbus\") pod \"global-pull-secret-syncer-jbwvd\" (UID: \"5981d2c3-3af8-4f97-b00f-eed942743c86\") " pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:02.559549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558404 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/754c2d81-3f3e-44d4-8589-fec36d904b15-host\") pod \"node-ca-pn5jp\" (UID: \"754c2d81-3f3e-44d4-8589-fec36d904b15\") " pod="openshift-image-registry/node-ca-pn5jp" Jul 23 00:08:02.559549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558439 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-run-openvswitch\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.559549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558463 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.559549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558465 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8507188-7090-4b88-940f-cf6f13963aea-system-cni-dir\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.559549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558489 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.559549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558517 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-sysctl-conf\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.559549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558548 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-kubelet\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.559549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558571 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.559549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558607 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-modprobe-d\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.559549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558634 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-sysconfig\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.559549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558662 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5qfq\" (UniqueName: \"kubernetes.io/projected/225d2d8c-3a75-47f8-9943-2f67e3c62e8b-kube-api-access-j5qfq\") pod \"node-resolver-t75zr\" (UID: \"225d2d8c-3a75-47f8-9943-2f67e3c62e8b\") " pod="openshift-dns/node-resolver-t75zr" Jul 23 00:08:02.559549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558686 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5981d2c3-3af8-4f97-b00f-eed942743c86-kubelet-config\") pod \"global-pull-secret-syncer-jbwvd\" (UID: \"5981d2c3-3af8-4f97-b00f-eed942743c86\") " pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:02.559549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558710 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f8507188-7090-4b88-940f-cf6f13963aea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.559549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558738 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mql7f\" (UniqueName: \"kubernetes.io/projected/907325cd-64b7-4241-8969-a32a19bab628-kube-api-access-mql7f\") pod \"network-metrics-daemon-czq2t\" (UID: \"907325cd-64b7-4241-8969-a32a19bab628\") " pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:02.559549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558762 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-sysctl-d\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558818 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-var-lib-openvswitch\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558833 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-run-systemd\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558847 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbd6z\" (UniqueName: \"kubernetes.io/projected/3626e0e2-750a-4540-af36-dba6f4802240-kube-api-access-fbd6z\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558878 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-socket-dir\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558899 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-node-log\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558906 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57ss4\" (UniqueName: \"kubernetes.io/projected/f8507188-7090-4b88-940f-cf6f13963aea-kube-api-access-57ss4\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558932 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-run\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558940 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-cni-bin\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558956 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f8507188-7090-4b88-940f-cf6f13963aea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558992 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-etc-openvswitch\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.558998 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/225d2d8c-3a75-47f8-9943-2f67e3c62e8b-hosts-file\") pod \"node-resolver-t75zr\" (UID: \"225d2d8c-3a75-47f8-9943-2f67e3c62e8b\") " pod="openshift-dns/node-resolver-t75zr" Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559019 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-etc-selinux\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559045 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-sys\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559071 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-lib-modules\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559095 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-sys-fs\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559114 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-sysctl-conf\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:02.559170 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:02.560377 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559188 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-run-ovn\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.561309 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:02.559238 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret podName:5981d2c3-3af8-4f97-b00f-eed942743c86 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:03.059217 +0000 UTC m=+3.019955617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret") pod "global-pull-secret-syncer-jbwvd" (UID: "5981d2c3-3af8-4f97-b00f-eed942743c86") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:02.561309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559241 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-registration-dir\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.561309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559282 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76c4a8e8-f0b1-4468-996d-dae6883fdca4-konnectivity-ca\") pod \"konnectivity-agent-2x8c9\" (UID: \"76c4a8e8-f0b1-4468-996d-dae6883fdca4\") " pod="kube-system/konnectivity-agent-2x8c9" Jul 23 00:08:02.561309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559290 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8507188-7090-4b88-940f-cf6f13963aea-cnibin\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.561309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559344 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f8507188-7090-4b88-940f-cf6f13963aea-os-release\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.561309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559546 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-var-lib-openvswitch\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.561309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559602 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.561309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559635 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3626e0e2-750a-4540-af36-dba6f4802240-ovnkube-script-lib\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.561309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559642 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-sysctl-d\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.561309 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:02.559663 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:02.561309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559669 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8507188-7090-4b88-940f-cf6f13963aea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.561309 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:02.559708 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs podName:907325cd-64b7-4241-8969-a32a19bab628 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:03.059693709 +0000 UTC m=+3.020432332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs") pod "network-metrics-daemon-czq2t" (UID: "907325cd-64b7-4241-8969-a32a19bab628") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:02.561309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559743 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-systemd-units\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.561309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559765 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-etc-selinux\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.561309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559821 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5981d2c3-3af8-4f97-b00f-eed942743c86-kubelet-config\") pod \"global-pull-secret-syncer-jbwvd\" (UID: \"5981d2c3-3af8-4f97-b00f-eed942743c86\") " pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:02.561309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559858 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-run\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.562164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559864 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-sys\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.562164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559875 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/225d2d8c-3a75-47f8-9943-2f67e3c62e8b-tmp-dir\") pod \"node-resolver-t75zr\" (UID: \"225d2d8c-3a75-47f8-9943-2f67e3c62e8b\") " pod="openshift-dns/node-resolver-t75zr" Jul 23 00:08:02.562164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.559959 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-modprobe-d\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.562164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.560030 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-lib-modules\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.562164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.560071 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.562164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.560121 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5981d2c3-3af8-4f97-b00f-eed942743c86-dbus\") pod \"global-pull-secret-syncer-jbwvd\" (UID: \"5981d2c3-3af8-4f97-b00f-eed942743c86\") " pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:02.562164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.560153 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/754c2d81-3f3e-44d4-8589-fec36d904b15-host\") pod \"node-ca-pn5jp\" (UID: \"754c2d81-3f3e-44d4-8589-fec36d904b15\") " pod="openshift-image-registry/node-ca-pn5jp" Jul 23 00:08:02.562164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.560182 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-run-openvswitch\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.562164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.560210 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.562164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.560402 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-socket-dir\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.562164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.560423 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-etc-openvswitch\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.562164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.560446 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3626e0e2-750a-4540-af36-dba6f4802240-host-kubelet\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.562164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.560494 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-sysconfig\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.562164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.560498 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4546908a-9199-44a1-aae5-603ac8e91370-sys-fs\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.562164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.560579 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f8507188-7090-4b88-940f-cf6f13963aea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.562164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.560844 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3626e0e2-750a-4540-af36-dba6f4802240-env-overrides\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.562164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.560875 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f8507188-7090-4b88-940f-cf6f13963aea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.562974 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.561345 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76c4a8e8-f0b1-4468-996d-dae6883fdca4-agent-certs\") pod \"konnectivity-agent-2x8c9\" (UID: \"76c4a8e8-f0b1-4468-996d-dae6883fdca4\") " pod="kube-system/konnectivity-agent-2x8c9" Jul 23 00:08:02.562974 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.561630 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/06fa9895-13a2-45a5-bc6a-b618b99c5de4-etc-tuned\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.562974 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.561717 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3626e0e2-750a-4540-af36-dba6f4802240-ovn-node-metrics-cert\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.562974 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.562523 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06fa9895-13a2-45a5-bc6a-b618b99c5de4-tmp\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.565112 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:02.565092 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:08:02.565112 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:02.565113 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:08:02.565275 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:02.565127 2555 projected.go:196] Error preparing data for projected volume kube-api-access-sqsfw for pod openshift-network-diagnostics/network-check-target-zgb4z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:02.565275 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:02.565181 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw podName:00044a26-c2cd-42d4-98df-5b62a5084d8b nodeName:}" failed. No retries permitted until 2026-07-23 00:08:03.065167359 +0000 UTC m=+3.025905972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sqsfw" (UniqueName: "kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw") pod "network-check-target-zgb4z" (UID: "00044a26-c2cd-42d4-98df-5b62a5084d8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:02.566054 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.566021 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znx59\" (UniqueName: \"kubernetes.io/projected/754c2d81-3f3e-44d4-8589-fec36d904b15-kube-api-access-znx59\") pod \"node-ca-pn5jp\" (UID: \"754c2d81-3f3e-44d4-8589-fec36d904b15\") " pod="openshift-image-registry/node-ca-pn5jp" Jul 23 00:08:02.567285 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.567175 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv7j6\" (UniqueName: \"kubernetes.io/projected/4546908a-9199-44a1-aae5-603ac8e91370-kube-api-access-hv7j6\") pod \"aws-ebs-csi-driver-node-ccnt4\" (UID: \"4546908a-9199-44a1-aae5-603ac8e91370\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.568763 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.568737 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57ss4\" (UniqueName: \"kubernetes.io/projected/f8507188-7090-4b88-940f-cf6f13963aea-kube-api-access-57ss4\") pod \"multus-additional-cni-plugins-m9njl\" (UID: \"f8507188-7090-4b88-940f-cf6f13963aea\") " pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:02.569281 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.569254 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkr2s\" (UniqueName: \"kubernetes.io/projected/06fa9895-13a2-45a5-bc6a-b618b99c5de4-kube-api-access-mkr2s\") pod \"tuned-ssldm\" (UID: \"06fa9895-13a2-45a5-bc6a-b618b99c5de4\") " pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.569866 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.569761 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mql7f\" (UniqueName: \"kubernetes.io/projected/907325cd-64b7-4241-8969-a32a19bab628-kube-api-access-mql7f\") pod \"network-metrics-daemon-czq2t\" (UID: \"907325cd-64b7-4241-8969-a32a19bab628\") " pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:02.570656 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.570617 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbd6z\" (UniqueName: \"kubernetes.io/projected/3626e0e2-750a-4540-af36-dba6f4802240-kube-api-access-fbd6z\") pod \"ovnkube-node-d8rnh\" (UID: \"3626e0e2-750a-4540-af36-dba6f4802240\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.571128 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.571041 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5qfq\" (UniqueName: \"kubernetes.io/projected/225d2d8c-3a75-47f8-9943-2f67e3c62e8b-kube-api-access-j5qfq\") pod \"node-resolver-t75zr\" (UID: \"225d2d8c-3a75-47f8-9943-2f67e3c62e8b\") " pod="openshift-dns/node-resolver-t75zr" Jul 23 00:08:02.654635 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.654604 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b699m" Jul 23 00:08:02.664479 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.664445 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6nxnc" Jul 23 00:08:02.676406 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.676364 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:02.683464 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.683438 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" Jul 23 00:08:02.690283 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.690254 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ssldm" Jul 23 00:08:02.699094 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.699049 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pn5jp" Jul 23 00:08:02.707837 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.707779 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t75zr" Jul 23 00:08:02.716756 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.716735 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2x8c9" Jul 23 00:08:02.723558 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:02.723533 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m9njl" Jul 23 00:08:03.062238 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:03.062146 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs\") pod \"network-metrics-daemon-czq2t\" (UID: \"907325cd-64b7-4241-8969-a32a19bab628\") " pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:03.062238 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:03.062181 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret\") pod \"global-pull-secret-syncer-jbwvd\" (UID: \"5981d2c3-3af8-4f97-b00f-eed942743c86\") " pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:03.062446 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:03.062315 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:03.062446 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:03.062369 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs podName:907325cd-64b7-4241-8969-a32a19bab628 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:04.062355122 +0000 UTC m=+4.023093721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs") pod "network-metrics-daemon-czq2t" (UID: "907325cd-64b7-4241-8969-a32a19bab628") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:03.062446 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:03.062315 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:03.062557 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:03.062439 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret podName:5981d2c3-3af8-4f97-b00f-eed942743c86 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:04.062426825 +0000 UTC m=+4.023165422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret") pod "global-pull-secret-syncer-jbwvd" (UID: "5981d2c3-3af8-4f97-b00f-eed942743c86") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:03.162944 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:03.162905 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqsfw\" (UniqueName: \"kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw\") pod \"network-check-target-zgb4z\" (UID: \"00044a26-c2cd-42d4-98df-5b62a5084d8b\") " pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:03.163134 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:03.163063 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:08:03.163134 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:03.163085 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:08:03.163134 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:03.163100 2555 projected.go:196] Error preparing data for projected volume kube-api-access-sqsfw for pod openshift-network-diagnostics/network-check-target-zgb4z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:03.163275 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:03.163166 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw podName:00044a26-c2cd-42d4-98df-5b62a5084d8b nodeName:}" failed. No retries permitted until 2026-07-23 00:08:04.163148747 +0000 UTC m=+4.123887346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqsfw" (UniqueName: "kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw") pod "network-check-target-zgb4z" (UID: "00044a26-c2cd-42d4-98df-5b62a5084d8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:03.385561 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:03.385484 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 00:03:02 +0000 UTC" deadline="2028-01-25 07:40:59.23938697 +0000 UTC" Jul 23 00:08:03.385561 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:03.385514 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13231h32m55.853876311s" Jul 23 00:08:03.498067 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:03.496994 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:03.498067 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:03.497118 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:04.070871 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.070536 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs\") pod \"network-metrics-daemon-czq2t\" (UID: \"907325cd-64b7-4241-8969-a32a19bab628\") " pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:04.070871 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.070833 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret\") pod \"global-pull-secret-syncer-jbwvd\" (UID: \"5981d2c3-3af8-4f97-b00f-eed942743c86\") " pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:04.071079 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:04.071017 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:04.071132 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:04.071081 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret podName:5981d2c3-3af8-4f97-b00f-eed942743c86 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:06.071061855 +0000 UTC m=+6.031800471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret") pod "global-pull-secret-syncer-jbwvd" (UID: "5981d2c3-3af8-4f97-b00f-eed942743c86") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:04.071551 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:04.071491 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:04.071551 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:04.071547 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs podName:907325cd-64b7-4241-8969-a32a19bab628 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:06.071530771 +0000 UTC m=+6.032269384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs") pod "network-metrics-daemon-czq2t" (UID: "907325cd-64b7-4241-8969-a32a19bab628") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:04.174723 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.174675 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqsfw\" (UniqueName: \"kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw\") pod \"network-check-target-zgb4z\" (UID: \"00044a26-c2cd-42d4-98df-5b62a5084d8b\") " pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:04.174950 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:04.174930 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:08:04.175038 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:04.174958 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:08:04.175038 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:04.174970 2555 projected.go:196] Error preparing data for projected volume kube-api-access-sqsfw for pod openshift-network-diagnostics/network-check-target-zgb4z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:04.175038 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:04.175031 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw podName:00044a26-c2cd-42d4-98df-5b62a5084d8b nodeName:}" failed. No retries permitted until 2026-07-23 00:08:06.175013604 +0000 UTC m=+6.135752214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqsfw" (UniqueName: "kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw") pod "network-check-target-zgb4z" (UID: "00044a26-c2cd-42d4-98df-5b62a5084d8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:04.300796 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.300760 2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:08:04.386160 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.386019 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 00:03:02 +0000 UTC" deadline="2027-12-21 01:12:16.471906077 +0000 UTC" Jul 23 00:08:04.386160 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.386083 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12385h4m12.085827222s" Jul 23 00:08:04.500217 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.499559 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:04.500217 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:04.499694 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:04.500217 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.499776 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:04.500217 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:04.499850 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:04.531223 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.531113 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ssldm" event={"ID":"06fa9895-13a2-45a5-bc6a-b618b99c5de4","Type":"ContainerStarted","Data":"80583bc137f11fcd68f19198ccd2b278c79a086ce5404dcbca69a5e244c44d83"} Jul 23 00:08:04.544277 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.544205 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9njl" event={"ID":"f8507188-7090-4b88-940f-cf6f13963aea","Type":"ContainerStarted","Data":"30a1838dca646e28432e110ec6be897b6423030138e189ae124cf184b310eb2d"} Jul 23 00:08:04.556647 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.555613 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal" event={"ID":"6c386b32f5b824bf53480b029cd79f9a","Type":"ContainerStarted","Data":"d2a0835f213195c23bd0e478d70d452efaffafc0d25f8e06e5001f36eafc0189"} Jul 23 00:08:04.570901 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.570739 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-124.ec2.internal" podStartSLOduration=3.570719112 podStartE2EDuration="3.570719112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:04.569587215 +0000 UTC m=+4.530325834" watchObservedRunningTime="2026-07-23 00:08:04.570719112 +0000 UTC m=+4.531457734" Jul 23 00:08:04.570901 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.570842 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b699m" event={"ID":"60bc3129-fc54-4d47-a7b4-834eb559fd37","Type":"ContainerStarted","Data":"fc134833f55b6ab7b989c7d131186c770d5980fa4966124cdb332531fdccf395"} Jul 23 00:08:04.574814 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.574764 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pn5jp" event={"ID":"754c2d81-3f3e-44d4-8589-fec36d904b15","Type":"ContainerStarted","Data":"95e3dd4447feeb2f81916221ee2dc471b97176ca37ea4e95a00459131f1a6fc5"} Jul 23 00:08:04.582412 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.582369 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" event={"ID":"3626e0e2-750a-4540-af36-dba6f4802240","Type":"ContainerStarted","Data":"eb911cd4e1522362009180c31288016ba80783c846200f261003e2542295bd14"} Jul 23 00:08:04.586314 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.586286 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6nxnc" event={"ID":"54398313-28cf-4172-a8d3-dcc8ea7ca95b","Type":"ContainerStarted","Data":"93c30d285c14a82403ac58c3412cec6c7ba2cf79d4f68e00dfdbb7eb17e5370e"} Jul 23 00:08:04.597904 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.597872 2555 generic.go:356] "Generic (PLEG): container finished" podID="524c78bd725b7ef91dc2abac8d4936d9" containerID="7153a6ba243e7f9a3fee5ffd2ddac0942725434e0598133bec2e62955edf069d" exitCode=0 Jul 23 00:08:04.598166 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.598143 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal" event={"ID":"524c78bd725b7ef91dc2abac8d4936d9","Type":"ContainerDied","Data":"7153a6ba243e7f9a3fee5ffd2ddac0942725434e0598133bec2e62955edf069d"} Jul 23 00:08:04.604202 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.604177 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t75zr" event={"ID":"225d2d8c-3a75-47f8-9943-2f67e3c62e8b","Type":"ContainerStarted","Data":"0f190c3bf4d3f942a8489c4fda0f098d08e89592d97341cf4b869585b6ae4084"} Jul 23 00:08:04.617606 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.617509 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2x8c9" event={"ID":"76c4a8e8-f0b1-4468-996d-dae6883fdca4","Type":"ContainerStarted","Data":"76de07f293ef279a7f4b03d222988cf930cbd55d2a44f7b1be1d6a2734fa2ccb"} Jul 23 00:08:04.621796 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:04.621712 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" event={"ID":"4546908a-9199-44a1-aae5-603ac8e91370","Type":"ContainerStarted","Data":"5effb7423511bb09c97665b53bc04e8cb0bd3bff528f2b32df4f801a0f512e90"} Jul 23 00:08:05.496880 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:05.496414 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:05.496880 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:05.496542 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:05.641461 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:05.641343 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal" event={"ID":"524c78bd725b7ef91dc2abac8d4936d9","Type":"ContainerStarted","Data":"8d1aedb5bbd4ad03d7ae0e47b0504d0120160a1a1c68a1f5dfe99e9e6530153e"} Jul 23 00:08:05.656646 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:05.656481 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-124.ec2.internal" podStartSLOduration=4.656457769 podStartE2EDuration="4.656457769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:05.655524569 +0000 UTC m=+5.616263188" watchObservedRunningTime="2026-07-23 00:08:05.656457769 +0000 UTC m=+5.617196392" Jul 23 00:08:06.093201 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:06.092455 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs\") pod \"network-metrics-daemon-czq2t\" (UID: \"907325cd-64b7-4241-8969-a32a19bab628\") " pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:06.093201 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:06.092506 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret\") pod \"global-pull-secret-syncer-jbwvd\" (UID: \"5981d2c3-3af8-4f97-b00f-eed942743c86\") " pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:06.093201 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:06.092686 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:06.093201 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:06.092742 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret podName:5981d2c3-3af8-4f97-b00f-eed942743c86 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:10.092724552 +0000 UTC m=+10.053463156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret") pod "global-pull-secret-syncer-jbwvd" (UID: "5981d2c3-3af8-4f97-b00f-eed942743c86") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:06.093201 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:06.092802 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:06.093201 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:06.092859 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs podName:907325cd-64b7-4241-8969-a32a19bab628 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:10.092847426 +0000 UTC m=+10.053586030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs") pod "network-metrics-daemon-czq2t" (UID: "907325cd-64b7-4241-8969-a32a19bab628") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:06.196913 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:06.196874 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqsfw\" (UniqueName: \"kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw\") pod \"network-check-target-zgb4z\" (UID: \"00044a26-c2cd-42d4-98df-5b62a5084d8b\") " pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:06.197096 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:06.197052 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:08:06.197096 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:06.197078 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:08:06.197096 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:06.197092 2555 projected.go:196] Error preparing data for projected volume kube-api-access-sqsfw for pod openshift-network-diagnostics/network-check-target-zgb4z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:06.197255 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:06.197159 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw podName:00044a26-c2cd-42d4-98df-5b62a5084d8b nodeName:}" failed. No retries permitted until 2026-07-23 00:08:10.197132788 +0000 UTC m=+10.157871390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqsfw" (UniqueName: "kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw") pod "network-check-target-zgb4z" (UID: "00044a26-c2cd-42d4-98df-5b62a5084d8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:06.497291 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:06.497248 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:06.497749 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:06.497385 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:06.497749 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:06.497720 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:06.497876 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:06.497845 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:07.497178 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:07.497146 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:07.497348 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:07.497281 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:08.497719 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:08.497179 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:08.497719 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:08.497218 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:08.497719 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:08.497334 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:08.498262 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:08.497750 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:09.496632 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:09.496602 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:09.496845 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:09.496731 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:10.129144 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:10.129111 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs\") pod \"network-metrics-daemon-czq2t\" (UID: \"907325cd-64b7-4241-8969-a32a19bab628\") " pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:10.129621 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:10.129151 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret\") pod \"global-pull-secret-syncer-jbwvd\" (UID: \"5981d2c3-3af8-4f97-b00f-eed942743c86\") " pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:10.129621 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:10.129279 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:10.129621 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:10.129360 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret podName:5981d2c3-3af8-4f97-b00f-eed942743c86 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:18.129340108 +0000 UTC m=+18.090078707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret") pod "global-pull-secret-syncer-jbwvd" (UID: "5981d2c3-3af8-4f97-b00f-eed942743c86") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:10.129621 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:10.129279 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:10.129621 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:10.129527 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs podName:907325cd-64b7-4241-8969-a32a19bab628 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:18.129501839 +0000 UTC m=+18.090240442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs") pod "network-metrics-daemon-czq2t" (UID: "907325cd-64b7-4241-8969-a32a19bab628") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:10.230476 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:10.230413 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqsfw\" (UniqueName: \"kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw\") pod \"network-check-target-zgb4z\" (UID: \"00044a26-c2cd-42d4-98df-5b62a5084d8b\") " pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:10.230643 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:10.230567 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:08:10.230643 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:10.230591 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:08:10.230643 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:10.230603 2555 projected.go:196] Error preparing data for projected volume kube-api-access-sqsfw for pod openshift-network-diagnostics/network-check-target-zgb4z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:10.230814 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:10.230664 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw podName:00044a26-c2cd-42d4-98df-5b62a5084d8b nodeName:}" failed. No retries permitted until 2026-07-23 00:08:18.230644821 +0000 UTC m=+18.191383434 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqsfw" (UniqueName: "kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw") pod "network-check-target-zgb4z" (UID: "00044a26-c2cd-42d4-98df-5b62a5084d8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:10.497736 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:10.497704 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:10.497914 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:10.497828 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:10.497914 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:10.497873 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:10.498063 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:10.498020 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:11.497107 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:11.497062 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:11.497536 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:11.497201 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:12.497423 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:12.496876 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:12.497423 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:12.497025 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:12.498176 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:12.498029 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:12.498176 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:12.498136 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:13.497222 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:13.497188 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:13.497384 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:13.497299 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:14.497347 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:14.497308 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:14.497812 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:14.497357 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:14.497812 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:14.497479 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:14.497812 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:14.497625 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:15.497249 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:15.497206 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:15.497486 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:15.497344 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:16.497241 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:16.497203 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:16.497438 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:16.497203 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:16.497438 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:16.497351 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:16.497856 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:16.497447 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:17.496223 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:17.496187 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:17.496545 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:17.496306 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:18.191125 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:18.191076 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs\") pod \"network-metrics-daemon-czq2t\" (UID: \"907325cd-64b7-4241-8969-a32a19bab628\") " pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:18.191701 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:18.191138 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret\") pod \"global-pull-secret-syncer-jbwvd\" (UID: \"5981d2c3-3af8-4f97-b00f-eed942743c86\") " pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:18.191701 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:18.191239 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:18.191701 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:18.191254 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:18.191701 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:18.191303 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs podName:907325cd-64b7-4241-8969-a32a19bab628 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:34.191283376 +0000 UTC m=+34.152021979 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs") pod "network-metrics-daemon-czq2t" (UID: "907325cd-64b7-4241-8969-a32a19bab628") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:18.191701 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:18.191319 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret podName:5981d2c3-3af8-4f97-b00f-eed942743c86 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:34.191312722 +0000 UTC m=+34.152051319 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret") pod "global-pull-secret-syncer-jbwvd" (UID: "5981d2c3-3af8-4f97-b00f-eed942743c86") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:18.292024 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:18.291993 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqsfw\" (UniqueName: \"kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw\") pod \"network-check-target-zgb4z\" (UID: \"00044a26-c2cd-42d4-98df-5b62a5084d8b\") " pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:18.292187 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:18.292132 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:08:18.292187 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:18.292154 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:08:18.292187 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:18.292165 2555 projected.go:196] Error preparing data for projected volume kube-api-access-sqsfw for pod openshift-network-diagnostics/network-check-target-zgb4z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:18.292302 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:18.292215 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw podName:00044a26-c2cd-42d4-98df-5b62a5084d8b nodeName:}" failed. No retries permitted until 2026-07-23 00:08:34.292201886 +0000 UTC m=+34.252940482 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqsfw" (UniqueName: "kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw") pod "network-check-target-zgb4z" (UID: "00044a26-c2cd-42d4-98df-5b62a5084d8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:18.496651 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:18.496563 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:18.496820 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:18.496573 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:18.496820 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:18.496717 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:18.496820 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:18.496790 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:19.496966 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:19.496926 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:19.497427 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:19.497055 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:20.497087 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:20.497052 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:20.497586 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:20.497140 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:20.497586 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:20.497149 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:20.497586 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:20.497286 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:21.496780 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:21.496738 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:21.496932 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:21.496871 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:22.497413 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.497143 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:22.497879 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:22.497546 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:22.497879 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.497750 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:22.498552 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:22.498491 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:22.677041 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.676806 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t75zr" event={"ID":"225d2d8c-3a75-47f8-9943-2f67e3c62e8b","Type":"ContainerStarted","Data":"e1360dd68eb138d94e1b3178437202ade4a88b513cffb398e453d1dd657d7b2c"} Jul 23 00:08:22.678571 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.678540 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2x8c9" event={"ID":"76c4a8e8-f0b1-4468-996d-dae6883fdca4","Type":"ContainerStarted","Data":"053df0319267caa268d65a9251e69d68aaef395662b25739bf19a03f92a7c650"} Jul 23 00:08:22.680077 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.680049 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" event={"ID":"4546908a-9199-44a1-aae5-603ac8e91370","Type":"ContainerStarted","Data":"f76491dca92f5b168bb34861affd0e0fce000f83b1e7c1e77076fe327cd78e16"} Jul 23 00:08:22.681630 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.681595 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ssldm" event={"ID":"06fa9895-13a2-45a5-bc6a-b618b99c5de4","Type":"ContainerStarted","Data":"742eb7eb168baf4ddd9f387dfe8e0bc9dc1d387c02ad8364be1c67c6eabc1a25"} Jul 23 00:08:22.683179 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.683146 2555 generic.go:356] "Generic (PLEG): container finished" podID="f8507188-7090-4b88-940f-cf6f13963aea" containerID="eeb0e1720da5683a875ac4c80514404c07efcd58e333eb4ebe3b7563afe51ac6" exitCode=0 Jul 23 00:08:22.683317 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.683238 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9njl" event={"ID":"f8507188-7090-4b88-940f-cf6f13963aea","Type":"ContainerDied","Data":"eeb0e1720da5683a875ac4c80514404c07efcd58e333eb4ebe3b7563afe51ac6"} Jul 23 00:08:22.684982 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.684951 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b699m" event={"ID":"60bc3129-fc54-4d47-a7b4-834eb559fd37","Type":"ContainerStarted","Data":"d92a61cb416706c5e9832f209b47e120ab19f5813631a66906d2c17effa4ed40"} Jul 23 00:08:22.686626 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.686530 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pn5jp" event={"ID":"754c2d81-3f3e-44d4-8589-fec36d904b15","Type":"ContainerStarted","Data":"9a79bd3a8c1e1cd1341f5c7f20a5c50740941c85c7b42bb517952dea66b051f9"} Jul 23 00:08:22.688401 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.688355 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" event={"ID":"3626e0e2-750a-4540-af36-dba6f4802240","Type":"ContainerStarted","Data":"9037b5a534e562d27c9712798442157b1a597982df9006825e86fb66a53e9700"} Jul 23 00:08:22.688492 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.688437 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" event={"ID":"3626e0e2-750a-4540-af36-dba6f4802240","Type":"ContainerStarted","Data":"fdbcc04b44f3fec2084b67ef79a41bf9d8a3dc04ca102e3e7f34a212f3df502f"} Jul 23 00:08:22.691298 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.691248 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t75zr" podStartSLOduration=3.93489 podStartE2EDuration="22.691228606s" totalImagesPullingTime="18.756338606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:22.690780787 +0000 UTC m=+22.651519409" watchObservedRunningTime="2026-07-23 00:08:22.691228606 +0000 UTC m=+22.651967225" Jul 23 00:08:22.704827 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.704780 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2x8c9" podStartSLOduration=3.957256416 podStartE2EDuration="22.704766405s" totalImagesPullingTime="18.747509989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:22.704356042 +0000 UTC m=+22.665094662" watchObservedRunningTime="2026-07-23 00:08:22.704766405 +0000 UTC m=+22.665505024" Jul 23 00:08:22.721363 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.721326 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ssldm" podStartSLOduration=3.969147398 podStartE2EDuration="22.721308618s" totalImagesPullingTime="18.75216122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:22.721288893 +0000 UTC m=+22.682027504" watchObservedRunningTime="2026-07-23 00:08:22.721308618 +0000 UTC m=+22.682047227" Jul 23 00:08:22.734084 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.734036 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pn5jp" podStartSLOduration=8.035179483 podStartE2EDuration="22.734020738s" totalImagesPullingTime="14.698841255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:22.733605838 +0000 UTC m=+22.694344458" watchObservedRunningTime="2026-07-23 00:08:22.734020738 +0000 UTC m=+22.694759357" Jul 23 00:08:22.770889 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.770827 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-b699m" podStartSLOduration=3.9711297610000003 podStartE2EDuration="22.770803126s" totalImagesPullingTime="18.799673365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:22.77042231 +0000 UTC m=+22.731160934" watchObservedRunningTime="2026-07-23 00:08:22.770803126 +0000 UTC m=+22.731541746" Jul 23 00:08:22.823714 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.823683 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:22.889086 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.889057 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:22.989643 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:22.989602 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:23.496529 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:23.496488 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:23.496727 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:23.496622 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:23.694817 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:23.694621 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" event={"ID":"3626e0e2-750a-4540-af36-dba6f4802240","Type":"ContainerStarted","Data":"d53abe13f0ee10af07f2167ad10e94b5c3236bb168984a099e660f9c365d7b5d"} Jul 23 00:08:23.694817 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:23.694722 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" event={"ID":"3626e0e2-750a-4540-af36-dba6f4802240","Type":"ContainerStarted","Data":"721c3e083893fc89bbcc24041cae297e6d028317701725b5dc2d42938b0e54dd"} Jul 23 00:08:23.694817 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:23.694771 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" event={"ID":"3626e0e2-750a-4540-af36-dba6f4802240","Type":"ContainerStarted","Data":"92b33be6a78d7d3a92ccf39d8bc514ca977e55e154ac4c401491e959f0c746d5"} Jul 23 00:08:23.694817 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:23.694818 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" event={"ID":"3626e0e2-750a-4540-af36-dba6f4802240","Type":"ContainerStarted","Data":"2d1439486e11e9ed97e01ff0564d3d738c45e184ed2a8cc1bb66c93f14efc030"} Jul 23 00:08:24.203157 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:24.203123 2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 00:08:24.413545 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:24.413438 2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T00:08:24.203142494Z","UUID":"6a34b404-babc-4e4b-8921-9f52eb269625","Handler":null,"Name":"","Endpoint":""} Jul 23 00:08:24.415300 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:24.415261 2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 00:08:24.415300 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:24.415303 2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 00:08:24.496501 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:24.496440 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:24.496743 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:24.496440 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:24.496743 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:24.496614 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:24.496743 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:24.496640 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:24.699874 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:24.699775 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" event={"ID":"4546908a-9199-44a1-aae5-603ac8e91370","Type":"ContainerStarted","Data":"6218ba7e6f93df54e088c5e962d8ed1f7417c865a5e86aa61ccfb40a067a04ae"} Jul 23 00:08:24.701977 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:24.701948 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6nxnc" event={"ID":"54398313-28cf-4172-a8d3-dcc8ea7ca95b","Type":"ContainerStarted","Data":"fce2bf1d55c1cded5f9fecc64cc8269cd94b56babb61327d6cbfca5c2a2e161b"} Jul 23 00:08:24.840897 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:24.840863 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:24.900470 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:24.900424 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:25.192132 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:25.192107 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:25.196052 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:25.196029 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2x8c9" Jul 23 00:08:25.196521 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:25.196502 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2x8c9" Jul 23 00:08:25.211735 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:25.211686 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6nxnc" podStartSLOduration=6.45839104 podStartE2EDuration="25.211671735s" totalImagesPullingTime="18.753280695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:24.716853151 +0000 UTC m=+24.677591770" watchObservedRunningTime="2026-07-23 00:08:25.211671735 +0000 UTC m=+25.172410353" Jul 23 00:08:25.293150 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:25.293103 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:25.496810 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:25.496770 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:25.496973 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:25.496883 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:25.553682 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:25.553650 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:25.607863 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:25.607832 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:25.708799 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:25.708766 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" event={"ID":"3626e0e2-750a-4540-af36-dba6f4802240","Type":"ContainerStarted","Data":"e9c35a8c84be50fe4e00a0e4bb5b64fc7895157f200fe7cfcebb6ca11b9cbce6"} Jul 23 00:08:25.709225 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:25.708855 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2x8c9" Jul 23 00:08:25.709689 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:25.709669 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2x8c9" Jul 23 00:08:25.770865 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:25.770785 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:26.496920 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:26.496868 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:26.497114 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:26.497027 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:26.497114 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:26.497085 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:26.497271 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:26.497244 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:27.496704 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:27.496548 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:27.497335 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:27.496785 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:27.715219 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:27.715186 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" event={"ID":"4546908a-9199-44a1-aae5-603ac8e91370","Type":"ContainerStarted","Data":"d46845665ecfd4abbedc678fff828c907776b51ef02d93c110f62c07a259a596"} Jul 23 00:08:27.716931 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:27.716906 2555 generic.go:356] "Generic (PLEG): container finished" podID="f8507188-7090-4b88-940f-cf6f13963aea" containerID="44c3ffa4c6a3eab4e10bc15b4c8764440d5721528d2c391cd89e55038a491e7c" exitCode=0 Jul 23 00:08:27.717052 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:27.716992 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9njl" event={"ID":"f8507188-7090-4b88-940f-cf6f13963aea","Type":"ContainerDied","Data":"44c3ffa4c6a3eab4e10bc15b4c8764440d5721528d2c391cd89e55038a491e7c"} Jul 23 00:08:27.720502 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:27.720479 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" event={"ID":"3626e0e2-750a-4540-af36-dba6f4802240","Type":"ContainerStarted","Data":"8a0f96e108ad799b6eafeaacf008a4b3bedaaf8cfd9dc8ca4d80e71ceb728be6"} Jul 23 00:08:27.720836 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:27.720816 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:27.720908 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:27.720889 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:27.720996 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:27.720986 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:27.732175 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:27.731941 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccnt4" podStartSLOduration=4.59349775 podStartE2EDuration="27.731922024s" totalImagesPullingTime="23.138424274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:27.731436833 +0000 UTC m=+27.692175455" watchObservedRunningTime="2026-07-23 00:08:27.731922024 +0000 UTC m=+27.692660644" Jul 23 00:08:27.736680 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:27.736661 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:27.736775 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:27.736750 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:27.796756 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:27.796689 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" podStartSLOduration=8.98933239 podStartE2EDuration="27.796673063s" totalImagesPullingTime="18.807340673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:27.795788948 +0000 UTC m=+27.756527566" watchObservedRunningTime="2026-07-23 00:08:27.796673063 +0000 UTC m=+27.757411681" Jul 23 00:08:28.497293 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:28.497258 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:28.497818 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:28.497366 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:28.497818 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:28.497419 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:28.497818 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:28.497535 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:29.226168 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:29.226135 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:29.327635 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:29.327605 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:29.426930 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:29.426899 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:29.493456 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:29.493342 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zgb4z"] Jul 23 00:08:29.493607 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:29.493491 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:29.493665 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:29.493594 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:29.494152 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:29.494129 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jbwvd"] Jul 23 00:08:29.494257 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:29.494215 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:29.494331 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:29.494311 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:29.494751 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:29.494730 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-czq2t"] Jul 23 00:08:29.494855 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:29.494802 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:29.494923 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:29.494877 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:29.726538 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:29.726506 2555 generic.go:356] "Generic (PLEG): container finished" podID="f8507188-7090-4b88-940f-cf6f13963aea" containerID="820f816ab1d88ddb981d1c17343bd003bd99eabb2055d3d108bf7aff4753a979" exitCode=0 Jul 23 00:08:29.726975 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:29.726574 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9njl" event={"ID":"f8507188-7090-4b88-940f-cf6f13963aea","Type":"ContainerDied","Data":"820f816ab1d88ddb981d1c17343bd003bd99eabb2055d3d108bf7aff4753a979"} Jul 23 00:08:30.255427 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:30.255379 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:30.312030 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:30.311987 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:31.497076 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:31.496906 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:31.497425 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:31.496950 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:31.497425 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:31.497156 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:31.497425 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:31.497227 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:31.497425 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:31.496955 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:31.497425 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:31.497298 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:31.732931 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:31.732893 2555 generic.go:356] "Generic (PLEG): container finished" podID="f8507188-7090-4b88-940f-cf6f13963aea" containerID="6fc3e359a22084cb1cacc3493def07da4a8f7ce677483202e5cc6639662c85b5" exitCode=0 Jul 23 00:08:31.733098 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:31.732955 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9njl" event={"ID":"f8507188-7090-4b88-940f-cf6f13963aea","Type":"ContainerDied","Data":"6fc3e359a22084cb1cacc3493def07da4a8f7ce677483202e5cc6639662c85b5"} Jul 23 00:08:33.197829 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:33.197759 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:33.497172 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:33.497089 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:33.497172 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:33.497130 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:33.497374 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:33.497219 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:33.497374 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:33.497220 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:33.497374 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:33.497340 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:33.497525 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:33.497438 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:34.219405 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:34.219312 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret\") pod \"global-pull-secret-syncer-jbwvd\" (UID: \"5981d2c3-3af8-4f97-b00f-eed942743c86\") " pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:34.219405 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:34.219373 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs\") pod \"network-metrics-daemon-czq2t\" (UID: \"907325cd-64b7-4241-8969-a32a19bab628\") " pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:34.219793 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:34.219475 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:34.219793 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:34.219476 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:34.219793 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:34.219530 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs podName:907325cd-64b7-4241-8969-a32a19bab628 nodeName:}" failed. No retries permitted until 2026-07-23 00:09:06.219514275 +0000 UTC m=+66.180252872 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs") pod "network-metrics-daemon-czq2t" (UID: "907325cd-64b7-4241-8969-a32a19bab628") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:34.219793 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:34.219543 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret podName:5981d2c3-3af8-4f97-b00f-eed942743c86 nodeName:}" failed. No retries permitted until 2026-07-23 00:09:06.219537131 +0000 UTC m=+66.180275727 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret") pod "global-pull-secret-syncer-jbwvd" (UID: "5981d2c3-3af8-4f97-b00f-eed942743c86") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:34.320532 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:34.320490 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqsfw\" (UniqueName: \"kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw\") pod \"network-check-target-zgb4z\" (UID: \"00044a26-c2cd-42d4-98df-5b62a5084d8b\") " pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:34.320701 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:34.320665 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:08:34.320701 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:34.320691 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:08:34.320701 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:34.320701 2555 projected.go:196] Error preparing data for projected volume kube-api-access-sqsfw for pod openshift-network-diagnostics/network-check-target-zgb4z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:34.320840 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:34.320766 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw podName:00044a26-c2cd-42d4-98df-5b62a5084d8b nodeName:}" failed. No retries permitted until 2026-07-23 00:09:06.32074661 +0000 UTC m=+66.281485222 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sqsfw" (UniqueName: "kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw") pod "network-check-target-zgb4z" (UID: "00044a26-c2cd-42d4-98df-5b62a5084d8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:35.496652 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.496622 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:35.497169 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.496662 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:35.497169 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.496662 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:35.497169 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:35.496745 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbwvd" podUID="5981d2c3-3af8-4f97-b00f-eed942743c86" Jul 23 00:08:35.497169 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:35.496845 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:08:35.497169 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:35.496938 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgb4z" podUID="00044a26-c2cd-42d4-98df-5b62a5084d8b" Jul 23 00:08:35.896914 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.896887 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-124.ec2.internal" event="NodeReady" Jul 23 00:08:35.897083 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.897070 2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 00:08:35.931313 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.931281 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5db59d744-b57zf"] Jul 23 00:08:35.955515 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.955486 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-f5wg8"] Jul 23 00:08:35.955650 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.955609 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:35.957936 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.957907 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 23 00:08:35.958059 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.957997 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-t8qkr\"" Jul 23 00:08:35.958145 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.958091 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 23 00:08:35.958145 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.958127 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 23 00:08:35.967476 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.967380 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 23 00:08:35.979949 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.979924 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qwwgq"] Jul 23 00:08:35.980069 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.980052 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f5wg8" Jul 23 00:08:35.982330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.982310 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 00:08:35.982420 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.982375 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 00:08:35.982420 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.982407 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 00:08:35.982514 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:35.982445 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-httr9\"" Jul 23 00:08:36.000570 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.000546 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5db59d744-b57zf"] Jul 23 00:08:36.000676 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.000583 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f5wg8"] Jul 23 00:08:36.000676 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.000654 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qwwgq" Jul 23 00:08:36.000748 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.000689 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qwwgq"] Jul 23 00:08:36.002905 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.002890 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 00:08:36.002980 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.002957 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fhhqg\"" Jul 23 00:08:36.003016 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.002994 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 00:08:36.004990 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.004965 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:36.113250 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.113223 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:36.133431 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.133404 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwn59\" (UniqueName: \"kubernetes.io/projected/359cf272-977d-4bcd-8e31-d2d45a8b440f-kube-api-access-gwn59\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:08:36.133535 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.133441 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e4b524a-e6ce-4247-bf26-81fda530d64d-cert\") pod \"ingress-canary-f5wg8\" (UID: \"5e4b524a-e6ce-4247-bf26-81fda530d64d\") " pod="openshift-ingress-canary/ingress-canary-f5wg8" Jul 23 00:08:36.133535 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.133470 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-certificates\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.133535 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.133491 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khsk7\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-kube-api-access-khsk7\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.133657 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.133553 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:08:36.133657 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.133579 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/359cf272-977d-4bcd-8e31-d2d45a8b440f-tmp-dir\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:08:36.133657 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.133604 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-bound-sa-token\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.133771 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.133697 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/359cf272-977d-4bcd-8e31-d2d45a8b440f-config-volume\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:08:36.133771 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.133716 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbl2x\" (UniqueName: \"kubernetes.io/projected/5e4b524a-e6ce-4247-bf26-81fda530d64d-kube-api-access-gbl2x\") pod \"ingress-canary-f5wg8\" (UID: \"5e4b524a-e6ce-4247-bf26-81fda530d64d\") " pod="openshift-ingress-canary/ingress-canary-f5wg8" Jul 23 00:08:36.133771 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.133731 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.133771 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.133759 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-ca-trust-extracted\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.133903 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.133789 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-image-registry-private-configuration\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.133903 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.133804 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-trusted-ca\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.133903 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.133824 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-installation-pull-secrets\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.182962 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.182932 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:36.234199 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.234159 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/359cf272-977d-4bcd-8e31-d2d45a8b440f-config-volume\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:08:36.234199 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.234199 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbl2x\" (UniqueName: \"kubernetes.io/projected/5e4b524a-e6ce-4247-bf26-81fda530d64d-kube-api-access-gbl2x\") pod \"ingress-canary-f5wg8\" (UID: \"5e4b524a-e6ce-4247-bf26-81fda530d64d\") " pod="openshift-ingress-canary/ingress-canary-f5wg8" Jul 23 00:08:36.234485 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.234220 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.234485 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.234240 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-ca-trust-extracted\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.234485 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:36.234333 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:08:36.234485 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:36.234352 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db59d744-b57zf: secret "image-registry-tls" not found Jul 23 00:08:36.234485 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.234382 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-image-registry-private-configuration\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.234485 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:36.234447 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls podName:0a31e563-70c6-441d-bd67-cc3dadb8b3eb nodeName:}" failed. No retries permitted until 2026-07-23 00:08:36.734425476 +0000 UTC m=+36.695164093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls") pod "image-registry-5db59d744-b57zf" (UID: "0a31e563-70c6-441d-bd67-cc3dadb8b3eb") : secret "image-registry-tls" not found Jul 23 00:08:36.234485 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.234482 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-trusted-ca\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.234819 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.234511 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-installation-pull-secrets\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.234819 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.234542 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwn59\" (UniqueName: \"kubernetes.io/projected/359cf272-977d-4bcd-8e31-d2d45a8b440f-kube-api-access-gwn59\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:08:36.234819 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.234563 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e4b524a-e6ce-4247-bf26-81fda530d64d-cert\") pod \"ingress-canary-f5wg8\" (UID: \"5e4b524a-e6ce-4247-bf26-81fda530d64d\") " pod="openshift-ingress-canary/ingress-canary-f5wg8" Jul 23 00:08:36.234819 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.234606 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-certificates\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.235009 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.234917 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/359cf272-977d-4bcd-8e31-d2d45a8b440f-config-volume\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:08:36.235009 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.234908 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khsk7\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-kube-api-access-khsk7\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.235009 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.234999 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:08:36.235143 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.235035 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/359cf272-977d-4bcd-8e31-d2d45a8b440f-tmp-dir\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:08:36.235143 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.235059 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-bound-sa-token\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.235143 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.235133 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-certificates\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.235255 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:36.235150 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:08:36.235255 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:36.235201 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls podName:359cf272-977d-4bcd-8e31-d2d45a8b440f nodeName:}" failed. No retries permitted until 2026-07-23 00:08:36.73518343 +0000 UTC m=+36.695922028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls") pod "dns-default-qwwgq" (UID: "359cf272-977d-4bcd-8e31-d2d45a8b440f") : secret "dns-default-metrics-tls" not found Jul 23 00:08:36.235383 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.235361 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/359cf272-977d-4bcd-8e31-d2d45a8b440f-tmp-dir\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:08:36.235672 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.235469 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-ca-trust-extracted\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.235672 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.235517 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-trusted-ca\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.238633 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.238615 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-installation-pull-secrets\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.238633 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.238628 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e4b524a-e6ce-4247-bf26-81fda530d64d-cert\") pod \"ingress-canary-f5wg8\" (UID: \"5e4b524a-e6ce-4247-bf26-81fda530d64d\") " pod="openshift-ingress-canary/ingress-canary-f5wg8" Jul 23 00:08:36.238757 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.238628 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-image-registry-private-configuration\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.242832 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.242813 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbl2x\" (UniqueName: \"kubernetes.io/projected/5e4b524a-e6ce-4247-bf26-81fda530d64d-kube-api-access-gbl2x\") pod \"ingress-canary-f5wg8\" (UID: \"5e4b524a-e6ce-4247-bf26-81fda530d64d\") " pod="openshift-ingress-canary/ingress-canary-f5wg8" Jul 23 00:08:36.243295 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.243260 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwn59\" (UniqueName: \"kubernetes.io/projected/359cf272-977d-4bcd-8e31-d2d45a8b440f-kube-api-access-gwn59\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:08:36.243681 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.243661 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khsk7\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-kube-api-access-khsk7\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.243853 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.243835 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-bound-sa-token\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.288703 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.288673 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f5wg8" Jul 23 00:08:36.443232 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.443051 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f5wg8"] Jul 23 00:08:36.448016 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:08:36.447984 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4b524a_e6ce_4247_bf26_81fda530d64d.slice/crio-44cdc57d35dda22dc691549a68b60163c1f8ac30e2b1ad9826c2313ba2c0c7d8 WatchSource:0}: Error finding container 44cdc57d35dda22dc691549a68b60163c1f8ac30e2b1ad9826c2313ba2c0c7d8: Status 404 returned error can't find the container with id 44cdc57d35dda22dc691549a68b60163c1f8ac30e2b1ad9826c2313ba2c0c7d8 Jul 23 00:08:36.739715 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.739688 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:08:36.740210 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.739781 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:36.740210 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:36.739840 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:08:36.740210 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:36.739867 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:08:36.740210 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:36.739878 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db59d744-b57zf: secret "image-registry-tls" not found Jul 23 00:08:36.740210 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:36.739913 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls podName:359cf272-977d-4bcd-8e31-d2d45a8b440f nodeName:}" failed. No retries permitted until 2026-07-23 00:08:37.739891878 +0000 UTC m=+37.700630492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls") pod "dns-default-qwwgq" (UID: "359cf272-977d-4bcd-8e31-d2d45a8b440f") : secret "dns-default-metrics-tls" not found Jul 23 00:08:36.740210 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:36.739927 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls podName:0a31e563-70c6-441d-bd67-cc3dadb8b3eb nodeName:}" failed. No retries permitted until 2026-07-23 00:08:37.739921439 +0000 UTC m=+37.700660037 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls") pod "image-registry-5db59d744-b57zf" (UID: "0a31e563-70c6-441d-bd67-cc3dadb8b3eb") : secret "image-registry-tls" not found Jul 23 00:08:36.746937 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.746912 2555 generic.go:356] "Generic (PLEG): container finished" podID="f8507188-7090-4b88-940f-cf6f13963aea" containerID="7b8af93ae835d70dc82aa7053e5acfce0e98b6b4a061df86d3018661baa8c89b" exitCode=0 Jul 23 00:08:36.747066 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.746992 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9njl" event={"ID":"f8507188-7090-4b88-940f-cf6f13963aea","Type":"ContainerDied","Data":"7b8af93ae835d70dc82aa7053e5acfce0e98b6b4a061df86d3018661baa8c89b"} Jul 23 00:08:36.748100 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:36.748081 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f5wg8" event={"ID":"5e4b524a-e6ce-4247-bf26-81fda530d64d","Type":"ContainerStarted","Data":"44cdc57d35dda22dc691549a68b60163c1f8ac30e2b1ad9826c2313ba2c0c7d8"} Jul 23 00:08:37.496955 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:37.496917 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:08:37.497148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:37.496963 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:08:37.497148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:37.496917 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:08:37.499641 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:37.499611 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 00:08:37.499784 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:37.499675 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 00:08:37.499784 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:37.499680 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 00:08:37.501163 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:37.501135 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d8bbv\"" Jul 23 00:08:37.501294 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:37.501200 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pff25\"" Jul 23 00:08:37.501294 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:37.501222 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 00:08:37.749652 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:37.749577 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:08:37.750050 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:37.749710 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:08:37.750050 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:37.749774 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls podName:359cf272-977d-4bcd-8e31-d2d45a8b440f nodeName:}" failed. No retries permitted until 2026-07-23 00:08:39.749758647 +0000 UTC m=+39.710497247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls") pod "dns-default-qwwgq" (UID: "359cf272-977d-4bcd-8e31-d2d45a8b440f") : secret "dns-default-metrics-tls" not found Jul 23 00:08:37.750050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:37.749808 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:37.750050 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:37.749889 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:08:37.750050 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:37.749899 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db59d744-b57zf: secret "image-registry-tls" not found Jul 23 00:08:37.750050 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:37.749936 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls podName:0a31e563-70c6-441d-bd67-cc3dadb8b3eb nodeName:}" failed. No retries permitted until 2026-07-23 00:08:39.749924812 +0000 UTC m=+39.710663410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls") pod "image-registry-5db59d744-b57zf" (UID: "0a31e563-70c6-441d-bd67-cc3dadb8b3eb") : secret "image-registry-tls" not found Jul 23 00:08:37.753804 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:37.753780 2555 generic.go:356] "Generic (PLEG): container finished" podID="f8507188-7090-4b88-940f-cf6f13963aea" containerID="5c04f8f0cf82d962894eecca4be9472637c15edaa872f67997d15e6db1cdee00" exitCode=0 Jul 23 00:08:37.753918 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:37.753811 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9njl" event={"ID":"f8507188-7090-4b88-940f-cf6f13963aea","Type":"ContainerDied","Data":"5c04f8f0cf82d962894eecca4be9472637c15edaa872f67997d15e6db1cdee00"} Jul 23 00:08:38.759833 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:38.759609 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9njl" event={"ID":"f8507188-7090-4b88-940f-cf6f13963aea","Type":"ContainerStarted","Data":"70ca4e4b2c1d3b396646f431921e064b808a4f5ebae7a5fb777f4291d608f64c"} Jul 23 00:08:38.779317 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:38.779280 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m9njl" podStartSLOduration=9.013863412 podStartE2EDuration="38.779263791s" totalImagesPullingTime="29.765400379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:38.778299277 +0000 UTC m=+38.739037896" watchObservedRunningTime="2026-07-23 00:08:38.779263791 +0000 UTC m=+38.740002407" Jul 23 00:08:38.899127 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:38.899090 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:38.999478 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:38.999372 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:39.544779 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:39.544747 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:39.597150 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:39.597114 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:39.762178 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:39.762155 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:39.762762 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:39.762202 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:08:39.762762 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:39.762296 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:08:39.762762 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:39.762299 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:08:39.762762 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:39.762357 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls podName:359cf272-977d-4bcd-8e31-d2d45a8b440f nodeName:}" failed. No retries permitted until 2026-07-23 00:08:43.762341961 +0000 UTC m=+43.723080559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls") pod "dns-default-qwwgq" (UID: "359cf272-977d-4bcd-8e31-d2d45a8b440f") : secret "dns-default-metrics-tls" not found Jul 23 00:08:39.762762 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:39.762362 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db59d744-b57zf: secret "image-registry-tls" not found Jul 23 00:08:39.762762 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:39.762441 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls podName:0a31e563-70c6-441d-bd67-cc3dadb8b3eb nodeName:}" failed. No retries permitted until 2026-07-23 00:08:43.762424922 +0000 UTC m=+43.723163522 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls") pod "image-registry-5db59d744-b57zf" (UID: "0a31e563-70c6-441d-bd67-cc3dadb8b3eb") : secret "image-registry-tls" not found Jul 23 00:08:39.763548 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:39.763526 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f5wg8" event={"ID":"5e4b524a-e6ce-4247-bf26-81fda530d64d","Type":"ContainerStarted","Data":"5ad3bda75cc4584a816cf3b6f3f07db08344a920bb6b178240604e0f4783011f"} Jul 23 00:08:39.781093 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:39.781053 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-f5wg8" podStartSLOduration=2.489384073 podStartE2EDuration="4.781040856s" totalImagesPullingTime="2.291656783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:39.780588663 +0000 UTC m=+39.741327282" watchObservedRunningTime="2026-07-23 00:08:39.781040856 +0000 UTC m=+39.741779475" Jul 23 00:08:41.711805 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:41.711767 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:41.812382 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:41.812347 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:43.789996 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:43.789956 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:43.789996 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:43.790002 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:08:43.790479 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:43.790113 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:08:43.790479 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:43.790139 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db59d744-b57zf: secret "image-registry-tls" not found Jul 23 00:08:43.790479 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:43.790116 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:08:43.790479 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:43.790206 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls podName:0a31e563-70c6-441d-bd67-cc3dadb8b3eb nodeName:}" failed. No retries permitted until 2026-07-23 00:08:51.790191119 +0000 UTC m=+51.750929717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls") pod "image-registry-5db59d744-b57zf" (UID: "0a31e563-70c6-441d-bd67-cc3dadb8b3eb") : secret "image-registry-tls" not found Jul 23 00:08:43.790479 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:43.790264 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls podName:359cf272-977d-4bcd-8e31-d2d45a8b440f nodeName:}" failed. No retries permitted until 2026-07-23 00:08:51.790246773 +0000 UTC m=+51.750985373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls") pod "dns-default-qwwgq" (UID: "359cf272-977d-4bcd-8e31-d2d45a8b440f") : secret "dns-default-metrics-tls" not found Jul 23 00:08:45.348104 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:45.348072 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:45.448770 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:45.448735 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:46.014827 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:46.014792 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:46.066927 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:46.066892 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:47.163914 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:47.163879 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:47.223426 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:47.223379 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:51.847949 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:51.847915 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:08:51.848361 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:51.847987 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:08:51.848361 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:51.848074 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:08:51.848361 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:51.848084 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db59d744-b57zf: secret "image-registry-tls" not found Jul 23 00:08:51.848361 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:51.848085 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:08:51.848361 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:51.848130 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls podName:0a31e563-70c6-441d-bd67-cc3dadb8b3eb nodeName:}" failed. No retries permitted until 2026-07-23 00:09:07.848117804 +0000 UTC m=+67.808856402 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls") pod "image-registry-5db59d744-b57zf" (UID: "0a31e563-70c6-441d-bd67-cc3dadb8b3eb") : secret "image-registry-tls" not found Jul 23 00:08:51.848361 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:08:51.848142 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls podName:359cf272-977d-4bcd-8e31-d2d45a8b440f nodeName:}" failed. No retries permitted until 2026-07-23 00:09:07.84813659 +0000 UTC m=+67.808875188 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls") pod "dns-default-qwwgq" (UID: "359cf272-977d-4bcd-8e31-d2d45a8b440f") : secret "dns-default-metrics-tls" not found Jul 23 00:08:59.736859 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:59.736825 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8rnh" Jul 23 00:08:59.905638 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:59.905601 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:08:59.973894 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:08:59.973863 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:09:06.249242 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.249202 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs\") pod \"network-metrics-daemon-czq2t\" (UID: \"907325cd-64b7-4241-8969-a32a19bab628\") " pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:09:06.249242 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.249244 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret\") pod \"global-pull-secret-syncer-jbwvd\" (UID: \"5981d2c3-3af8-4f97-b00f-eed942743c86\") " pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:09:06.251920 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.251902 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 00:09:06.252086 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.252068 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 00:09:06.259777 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:09:06.259757 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 23 00:09:06.259839 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:09:06.259823 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs podName:907325cd-64b7-4241-8969-a32a19bab628 nodeName:}" failed. No retries permitted until 2026-07-23 00:10:10.259806841 +0000 UTC m=+130.220545437 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs") pod "network-metrics-daemon-czq2t" (UID: "907325cd-64b7-4241-8969-a32a19bab628") : secret "metrics-daemon-secret" not found Jul 23 00:09:06.263397 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.263369 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5981d2c3-3af8-4f97-b00f-eed942743c86-original-pull-secret\") pod \"global-pull-secret-syncer-jbwvd\" (UID: \"5981d2c3-3af8-4f97-b00f-eed942743c86\") " pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:09:06.309578 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.309550 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbwvd" Jul 23 00:09:06.311761 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.311735 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:09:06.350432 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.350367 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqsfw\" (UniqueName: \"kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw\") pod \"network-check-target-zgb4z\" (UID: \"00044a26-c2cd-42d4-98df-5b62a5084d8b\") " pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:09:06.352995 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.352975 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 00:09:06.363474 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.363434 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 00:09:06.370471 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.370339 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:09:06.374753 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.374729 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqsfw\" (UniqueName: \"kubernetes.io/projected/00044a26-c2cd-42d4-98df-5b62a5084d8b-kube-api-access-sqsfw\") pod \"network-check-target-zgb4z\" (UID: \"00044a26-c2cd-42d4-98df-5b62a5084d8b\") " pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:09:06.434081 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.434050 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jbwvd"] Jul 23 00:09:06.437131 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:09:06.437105 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5981d2c3_3af8_4f97_b00f_eed942743c86.slice/crio-d56135d1cc85228c0ed082fa881c109bb1bf197795e034acb864308629e79dfb WatchSource:0}: Error finding container d56135d1cc85228c0ed082fa881c109bb1bf197795e034acb864308629e79dfb: Status 404 returned error can't find the container with id d56135d1cc85228c0ed082fa881c109bb1bf197795e034acb864308629e79dfb Jul 23 00:09:06.619779 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.619699 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d8bbv\"" Jul 23 00:09:06.627941 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.627920 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:09:06.750439 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.750412 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zgb4z"] Jul 23 00:09:06.753044 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:09:06.753018 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00044a26_c2cd_42d4_98df_5b62a5084d8b.slice/crio-bb0641d02bddcc903d4ab5304f10d93bf7b4e88a6925a28f6094b6c179dad099 WatchSource:0}: Error finding container bb0641d02bddcc903d4ab5304f10d93bf7b4e88a6925a28f6094b6c179dad099: Status 404 returned error can't find the container with id bb0641d02bddcc903d4ab5304f10d93bf7b4e88a6925a28f6094b6c179dad099 Jul 23 00:09:06.824344 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.824308 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zgb4z" event={"ID":"00044a26-c2cd-42d4-98df-5b62a5084d8b","Type":"ContainerStarted","Data":"bb0641d02bddcc903d4ab5304f10d93bf7b4e88a6925a28f6094b6c179dad099"} Jul 23 00:09:06.825353 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:06.825332 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jbwvd" event={"ID":"5981d2c3-3af8-4f97-b00f-eed942743c86","Type":"ContainerStarted","Data":"d56135d1cc85228c0ed082fa881c109bb1bf197795e034acb864308629e79dfb"} Jul 23 00:09:07.863407 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:07.863362 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:09:07.863838 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:07.863469 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:09:07.863838 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:09:07.863523 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:09:07.863838 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:09:07.863554 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:09:07.863838 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:09:07.863567 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db59d744-b57zf: secret "image-registry-tls" not found Jul 23 00:09:07.863838 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:09:07.863607 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls podName:359cf272-977d-4bcd-8e31-d2d45a8b440f nodeName:}" failed. No retries permitted until 2026-07-23 00:09:39.8635869 +0000 UTC m=+99.824325521 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls") pod "dns-default-qwwgq" (UID: "359cf272-977d-4bcd-8e31-d2d45a8b440f") : secret "dns-default-metrics-tls" not found Jul 23 00:09:07.863838 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:09:07.863622 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls podName:0a31e563-70c6-441d-bd67-cc3dadb8b3eb nodeName:}" failed. No retries permitted until 2026-07-23 00:09:39.863615739 +0000 UTC m=+99.824354335 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls") pod "image-registry-5db59d744-b57zf" (UID: "0a31e563-70c6-441d-bd67-cc3dadb8b3eb") : secret "image-registry-tls" not found Jul 23 00:09:11.840789 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:11.840694 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zgb4z" event={"ID":"00044a26-c2cd-42d4-98df-5b62a5084d8b","Type":"ContainerStarted","Data":"c90eca7e85dc228b1ad624dd5e44d4c109828c44992541d0c74a2cbaf6406f8e"} Jul 23 00:09:11.840789 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:11.840759 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:09:11.842260 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:11.842228 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jbwvd" event={"ID":"5981d2c3-3af8-4f97-b00f-eed942743c86","Type":"ContainerStarted","Data":"a815ff4d0bc5ae56b669b3584001ce3aab778edd6012f461b1f92b6850d50522"} Jul 23 00:09:11.855623 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:11.855569 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zgb4z" podStartSLOduration=67.035044914 podStartE2EDuration="1m11.855549845s" totalImagesPullingTime="4.820504931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:09:11.855261356 +0000 UTC m=+71.815999979" watchObservedRunningTime="2026-07-23 00:09:11.855549845 +0000 UTC m=+71.816288465" Jul 23 00:09:39.886650 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:39.886594 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:09:39.887081 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:39.886684 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:09:39.887081 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:09:39.886796 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:09:39.887081 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:09:39.886810 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db59d744-b57zf: secret "image-registry-tls" not found Jul 23 00:09:39.887081 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:09:39.886856 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls podName:0a31e563-70c6-441d-bd67-cc3dadb8b3eb nodeName:}" failed. No retries permitted until 2026-07-23 00:10:43.886842755 +0000 UTC m=+163.847581352 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls") pod "image-registry-5db59d744-b57zf" (UID: "0a31e563-70c6-441d-bd67-cc3dadb8b3eb") : secret "image-registry-tls" not found Jul 23 00:09:39.887081 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:09:39.886794 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:09:39.887081 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:09:39.886942 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls podName:359cf272-977d-4bcd-8e31-d2d45a8b440f nodeName:}" failed. No retries permitted until 2026-07-23 00:10:43.886927945 +0000 UTC m=+163.847666543 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls") pod "dns-default-qwwgq" (UID: "359cf272-977d-4bcd-8e31-d2d45a8b440f") : secret "dns-default-metrics-tls" not found Jul 23 00:09:42.847088 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:42.847057 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zgb4z" Jul 23 00:09:42.862519 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:09:42.862475 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jbwvd" podStartSLOduration=97.714555567 podStartE2EDuration="1m42.862462337s" totalImagesPullingTime="5.14790677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:09:11.870959784 +0000 UTC m=+71.831698403" watchObservedRunningTime="2026-07-23 00:09:42.862462337 +0000 UTC m=+102.823200955" Jul 23 00:10:10.295148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:10:10.295098 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs\") pod \"network-metrics-daemon-czq2t\" (UID: \"907325cd-64b7-4241-8969-a32a19bab628\") " pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:10:10.295688 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:10:10.295236 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 23 00:10:10.295688 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:10:10.295289 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs podName:907325cd-64b7-4241-8969-a32a19bab628 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:12.295275905 +0000 UTC m=+252.256014502 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs") pod "network-metrics-daemon-czq2t" (UID: "907325cd-64b7-4241-8969-a32a19bab628") : secret "metrics-daemon-secret" not found Jul 23 00:10:38.967699 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:10:38.967648 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5db59d744-b57zf" podUID="0a31e563-70c6-441d-bd67-cc3dadb8b3eb" Jul 23 00:10:39.008134 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:10:39.008094 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-qwwgq" podUID="359cf272-977d-4bcd-8e31-d2d45a8b440f" Jul 23 00:10:39.042459 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:10:39.042438 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:10:39.042573 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:10:39.042439 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qwwgq" Jul 23 00:10:40.523680 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:10:40.523632 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:10:43.923103 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:10:43.923065 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:10:43.923545 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:10:43.923131 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:10:43.923545 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:10:43.923216 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:10:43.923545 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:10:43.923229 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db59d744-b57zf: secret "image-registry-tls" not found Jul 23 00:10:43.923545 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:10:43.923275 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls podName:0a31e563-70c6-441d-bd67-cc3dadb8b3eb nodeName:}" failed. No retries permitted until 2026-07-23 00:12:45.923260856 +0000 UTC m=+285.883999454 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls") pod "image-registry-5db59d744-b57zf" (UID: "0a31e563-70c6-441d-bd67-cc3dadb8b3eb") : secret "image-registry-tls" not found Jul 23 00:10:43.923545 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:10:43.923215 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:10:43.923545 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:10:43.923341 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls podName:359cf272-977d-4bcd-8e31-d2d45a8b440f nodeName:}" failed. No retries permitted until 2026-07-23 00:12:45.923325646 +0000 UTC m=+285.884064242 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls") pod "dns-default-qwwgq" (UID: "359cf272-977d-4bcd-8e31-d2d45a8b440f") : secret "dns-default-metrics-tls" not found Jul 23 00:10:52.496442 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:10:52.496341 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:12:01.443585 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.443543 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-98rlm"] Jul 23 00:12:01.446534 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.446515 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml"] Jul 23 00:12:01.446664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.446647 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:01.448883 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.448860 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 23 00:12:01.449315 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.449298 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" Jul 23 00:12:01.449656 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.449638 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 23 00:12:01.450116 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.450098 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 23 00:12:01.450448 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.450435 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:12:01.451533 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.451518 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x27j9\"" Jul 23 00:12:01.453754 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.453734 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 00:12:01.453861 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.453803 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-n5xkn\"" Jul 23 00:12:01.453927 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.453908 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 23 00:12:01.453975 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.453926 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 00:12:01.453975 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.453939 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 23 00:12:01.456950 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.456930 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 23 00:12:01.468921 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.468893 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml"] Jul 23 00:12:01.469625 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.469607 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-98rlm"] Jul 23 00:12:01.544666 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.544637 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm"] Jul 23 00:12:01.547446 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.547431 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm" Jul 23 00:12:01.550419 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.550383 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 23 00:12:01.550419 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.550412 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:12:01.550598 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.550418 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 23 00:12:01.550598 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.550515 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-cd4dx\"" Jul 23 00:12:01.550598 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.550381 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 23 00:12:01.559313 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.559291 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm"] Jul 23 00:12:01.560523 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.560502 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19897756-2813-4980-865c-1bbb2012aa33-trusted-ca\") pod \"console-operator-565b7b569d-98rlm\" (UID: \"19897756-2813-4980-865c-1bbb2012aa33\") " pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:01.560641 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.560551 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwpc4\" (UniqueName: \"kubernetes.io/projected/165778da-7482-49cf-948f-5440431f12e1-kube-api-access-mwpc4\") pod \"cluster-monitoring-operator-76cf8966f8-qm5ml\" (UID: \"165778da-7482-49cf-948f-5440431f12e1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" Jul 23 00:12:01.560641 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.560576 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/165778da-7482-49cf-948f-5440431f12e1-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-qm5ml\" (UID: \"165778da-7482-49cf-948f-5440431f12e1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" Jul 23 00:12:01.560641 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.560609 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19897756-2813-4980-865c-1bbb2012aa33-config\") pod \"console-operator-565b7b569d-98rlm\" (UID: \"19897756-2813-4980-865c-1bbb2012aa33\") " pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:01.560641 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.560635 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qm5ml\" (UID: \"165778da-7482-49cf-948f-5440431f12e1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" Jul 23 00:12:01.560996 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.560653 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/19897756-2813-4980-865c-1bbb2012aa33-serving-cert\") pod \"console-operator-565b7b569d-98rlm\" (UID: \"19897756-2813-4980-865c-1bbb2012aa33\") " pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:01.560996 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.560668 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m7z6\" (UniqueName: \"kubernetes.io/projected/19897756-2813-4980-865c-1bbb2012aa33-kube-api-access-8m7z6\") pod \"console-operator-565b7b569d-98rlm\" (UID: \"19897756-2813-4980-865c-1bbb2012aa33\") " pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:01.661699 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.661664 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6khp2\" (UniqueName: \"kubernetes.io/projected/73b367b0-9408-4fcd-9535-cb198cfd7e34-kube-api-access-6khp2\") pod \"kube-storage-version-migrator-operator-7658d9b749-r7vwm\" (UID: \"73b367b0-9408-4fcd-9535-cb198cfd7e34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm" Jul 23 00:12:01.661874 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.661706 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qm5ml\" (UID: \"165778da-7482-49cf-948f-5440431f12e1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" Jul 23 00:12:01.661874 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.661728 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/19897756-2813-4980-865c-1bbb2012aa33-serving-cert\") pod \"console-operator-565b7b569d-98rlm\" (UID: \"19897756-2813-4980-865c-1bbb2012aa33\") " pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:01.661874 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.661752 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m7z6\" (UniqueName: \"kubernetes.io/projected/19897756-2813-4980-865c-1bbb2012aa33-kube-api-access-8m7z6\") pod \"console-operator-565b7b569d-98rlm\" (UID: \"19897756-2813-4980-865c-1bbb2012aa33\") " pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:01.661874 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.661805 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19897756-2813-4980-865c-1bbb2012aa33-trusted-ca\") pod \"console-operator-565b7b569d-98rlm\" (UID: \"19897756-2813-4980-865c-1bbb2012aa33\") " pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:01.661874 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:01.661831 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 00:12:01.661874 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.661852 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73b367b0-9408-4fcd-9535-cb198cfd7e34-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-r7vwm\" (UID: \"73b367b0-9408-4fcd-9535-cb198cfd7e34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm" Jul 23 00:12:01.662188 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.661890 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwpc4\" (UniqueName: \"kubernetes.io/projected/165778da-7482-49cf-948f-5440431f12e1-kube-api-access-mwpc4\") pod \"cluster-monitoring-operator-76cf8966f8-qm5ml\" (UID: \"165778da-7482-49cf-948f-5440431f12e1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" Jul 23 00:12:01.662188 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:01.661907 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls podName:165778da-7482-49cf-948f-5440431f12e1 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:02.161885004 +0000 UTC m=+242.122623621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-qm5ml" (UID: "165778da-7482-49cf-948f-5440431f12e1") : secret "cluster-monitoring-operator-tls" not found Jul 23 00:12:01.662188 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.661945 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/165778da-7482-49cf-948f-5440431f12e1-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-qm5ml\" (UID: \"165778da-7482-49cf-948f-5440431f12e1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" Jul 23 00:12:01.662188 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.661982 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19897756-2813-4980-865c-1bbb2012aa33-config\") pod \"console-operator-565b7b569d-98rlm\" (UID: \"19897756-2813-4980-865c-1bbb2012aa33\") " pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:01.662188 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.662013 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73b367b0-9408-4fcd-9535-cb198cfd7e34-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-r7vwm\" (UID: \"73b367b0-9408-4fcd-9535-cb198cfd7e34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm" Jul 23 00:12:01.662682 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.662658 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19897756-2813-4980-865c-1bbb2012aa33-config\") pod \"console-operator-565b7b569d-98rlm\" (UID: \"19897756-2813-4980-865c-1bbb2012aa33\") " pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:01.662778 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.662699 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/165778da-7482-49cf-948f-5440431f12e1-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-qm5ml\" (UID: \"165778da-7482-49cf-948f-5440431f12e1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" Jul 23 00:12:01.662913 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.662894 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19897756-2813-4980-865c-1bbb2012aa33-trusted-ca\") pod \"console-operator-565b7b569d-98rlm\" (UID: \"19897756-2813-4980-865c-1bbb2012aa33\") " pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:01.664091 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.664076 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/19897756-2813-4980-865c-1bbb2012aa33-serving-cert\") pod \"console-operator-565b7b569d-98rlm\" (UID: \"19897756-2813-4980-865c-1bbb2012aa33\") " pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:01.675370 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.671987 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwpc4\" (UniqueName: \"kubernetes.io/projected/165778da-7482-49cf-948f-5440431f12e1-kube-api-access-mwpc4\") pod \"cluster-monitoring-operator-76cf8966f8-qm5ml\" (UID: \"165778da-7482-49cf-948f-5440431f12e1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" Jul 23 00:12:01.675370 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.672014 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m7z6\" (UniqueName: \"kubernetes.io/projected/19897756-2813-4980-865c-1bbb2012aa33-kube-api-access-8m7z6\") pod \"console-operator-565b7b569d-98rlm\" (UID: \"19897756-2813-4980-865c-1bbb2012aa33\") " pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:01.716619 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.716532 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:01.757379 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.757346 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:01.762577 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.762556 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73b367b0-9408-4fcd-9535-cb198cfd7e34-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-r7vwm\" (UID: \"73b367b0-9408-4fcd-9535-cb198cfd7e34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm" Jul 23 00:12:01.762652 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.762591 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6khp2\" (UniqueName: \"kubernetes.io/projected/73b367b0-9408-4fcd-9535-cb198cfd7e34-kube-api-access-6khp2\") pod \"kube-storage-version-migrator-operator-7658d9b749-r7vwm\" (UID: \"73b367b0-9408-4fcd-9535-cb198cfd7e34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm" Jul 23 00:12:01.762705 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.762653 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73b367b0-9408-4fcd-9535-cb198cfd7e34-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-r7vwm\" (UID: \"73b367b0-9408-4fcd-9535-cb198cfd7e34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm" Jul 23 00:12:01.763169 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.763145 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73b367b0-9408-4fcd-9535-cb198cfd7e34-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-r7vwm\" (UID: \"73b367b0-9408-4fcd-9535-cb198cfd7e34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm" Jul 23 00:12:01.764838 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.764815 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73b367b0-9408-4fcd-9535-cb198cfd7e34-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-r7vwm\" (UID: \"73b367b0-9408-4fcd-9535-cb198cfd7e34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm" Jul 23 00:12:01.770165 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.770140 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6khp2\" (UniqueName: \"kubernetes.io/projected/73b367b0-9408-4fcd-9535-cb198cfd7e34-kube-api-access-6khp2\") pod \"kube-storage-version-migrator-operator-7658d9b749-r7vwm\" (UID: \"73b367b0-9408-4fcd-9535-cb198cfd7e34\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm" Jul 23 00:12:01.787427 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.787380 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:01.855805 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.855780 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm" Jul 23 00:12:01.881890 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.881781 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-98rlm"] Jul 23 00:12:01.886577 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:12:01.886540 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19897756_2813_4980_865c_1bbb2012aa33.slice/crio-972af2339d91f5fc10df83f4f7be5a3dc39c2c6aa427dc4a29f0c1ffe0daff90 WatchSource:0}: Error finding container 972af2339d91f5fc10df83f4f7be5a3dc39c2c6aa427dc4a29f0c1ffe0daff90: Status 404 returned error can't find the container with id 972af2339d91f5fc10df83f4f7be5a3dc39c2c6aa427dc4a29f0c1ffe0daff90 Jul 23 00:12:01.977223 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:01.977139 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm"] Jul 23 00:12:01.980282 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:12:01.980259 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b367b0_9408_4fcd_9535_cb198cfd7e34.slice/crio-931efd8138071d5f535d203cf45c9d4203b2f81831c61c847731c1f08570462d WatchSource:0}: Error finding container 931efd8138071d5f535d203cf45c9d4203b2f81831c61c847731c1f08570462d: Status 404 returned error can't find the container with id 931efd8138071d5f535d203cf45c9d4203b2f81831c61c847731c1f08570462d Jul 23 00:12:02.166407 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:02.166366 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qm5ml\" (UID: \"165778da-7482-49cf-948f-5440431f12e1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" Jul 23 00:12:02.166602 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:02.166540 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 00:12:02.166664 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:02.166623 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls podName:165778da-7482-49cf-948f-5440431f12e1 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:03.166600519 +0000 UTC m=+243.127339129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-qm5ml" (UID: "165778da-7482-49cf-948f-5440431f12e1") : secret "cluster-monitoring-operator-tls" not found Jul 23 00:12:02.230863 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:02.230773 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-98rlm" event={"ID":"19897756-2813-4980-865c-1bbb2012aa33","Type":"ContainerStarted","Data":"972af2339d91f5fc10df83f4f7be5a3dc39c2c6aa427dc4a29f0c1ffe0daff90"} Jul 23 00:12:02.231727 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:02.231699 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm" event={"ID":"73b367b0-9408-4fcd-9535-cb198cfd7e34","Type":"ContainerStarted","Data":"931efd8138071d5f535d203cf45c9d4203b2f81831c61c847731c1f08570462d"} Jul 23 00:12:03.175735 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:03.175689 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qm5ml\" (UID: \"165778da-7482-49cf-948f-5440431f12e1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" Jul 23 00:12:03.176129 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:03.175842 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 00:12:03.176129 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:03.175921 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls podName:165778da-7482-49cf-948f-5440431f12e1 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:05.175905226 +0000 UTC m=+245.136643824 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-qm5ml" (UID: "165778da-7482-49cf-948f-5440431f12e1") : secret "cluster-monitoring-operator-tls" not found Jul 23 00:12:03.771231 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:03.771179 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:03.820622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:03.820550 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:04.903575 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:04.903538 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9db4t"] Jul 23 00:12:04.906709 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:04.906596 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:04.909138 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:04.909115 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 00:12:04.910221 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:04.910204 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5btmf\"" Jul 23 00:12:04.910364 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:04.910229 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 00:12:04.910364 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:04.910292 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 00:12:04.910468 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:04.910443 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 00:12:04.928656 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:04.928618 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9db4t"] Jul 23 00:12:04.942263 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:04.942175 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:04.988462 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:04.988416 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccd64\" (UniqueName: \"kubernetes.io/projected/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-api-access-ccd64\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:04.988658 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:04.988518 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c3cef2c-367d-4cea-bf38-e952f424ff69-crio-socket\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:04.988658 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:04.988551 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:04.988658 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:04.988621 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c3cef2c-367d-4cea-bf38-e952f424ff69-data-volume\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:04.988658 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:04.988645 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:05.004261 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.004231 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:05.089551 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.089513 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c3cef2c-367d-4cea-bf38-e952f424ff69-crio-socket\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:05.089693 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.089558 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:05.089693 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.089597 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c3cef2c-367d-4cea-bf38-e952f424ff69-data-volume\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:05.089693 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.089611 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:05.089693 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.089634 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccd64\" (UniqueName: \"kubernetes.io/projected/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-api-access-ccd64\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:05.089693 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.089640 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c3cef2c-367d-4cea-bf38-e952f424ff69-crio-socket\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:05.089941 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:05.089723 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:12:05.089941 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:05.089794 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm podName:1c3cef2c-367d-4cea-bf38-e952f424ff69 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:05.589775136 +0000 UTC m=+245.550513734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9db4t" (UID: "1c3cef2c-367d-4cea-bf38-e952f424ff69") : configmap "kube-rbac-proxy" not found Jul 23 00:12:05.089941 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:05.089729 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:12:05.089941 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:05.089854 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls podName:1c3cef2c-367d-4cea-bf38-e952f424ff69 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:05.589841402 +0000 UTC m=+245.550579999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9db4t" (UID: "1c3cef2c-367d-4cea-bf38-e952f424ff69") : secret "insights-runtime-extractor-tls" not found Jul 23 00:12:05.090146 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.089976 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c3cef2c-367d-4cea-bf38-e952f424ff69-data-volume\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:05.099668 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.099640 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccd64\" (UniqueName: \"kubernetes.io/projected/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-api-access-ccd64\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:05.190170 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.190134 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qm5ml\" (UID: \"165778da-7482-49cf-948f-5440431f12e1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" Jul 23 00:12:05.190351 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:05.190302 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 00:12:05.190429 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:05.190365 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls podName:165778da-7482-49cf-948f-5440431f12e1 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:09.190347901 +0000 UTC m=+249.151086502 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-qm5ml" (UID: "165778da-7482-49cf-948f-5440431f12e1") : secret "cluster-monitoring-operator-tls" not found Jul 23 00:12:05.241480 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.241378 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/0.log" Jul 23 00:12:05.241480 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.241437 2555 generic.go:356] "Generic (PLEG): container finished" podID="19897756-2813-4980-865c-1bbb2012aa33" containerID="45f459d7340dae6e14a3cdcabc82321380f9a9e9f5438f441e1bbc2fb5959b44" exitCode=255 Jul 23 00:12:05.241702 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.241473 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-98rlm" event={"ID":"19897756-2813-4980-865c-1bbb2012aa33","Type":"ContainerDied","Data":"45f459d7340dae6e14a3cdcabc82321380f9a9e9f5438f441e1bbc2fb5959b44"} Jul 23 00:12:05.241774 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.241757 2555 scope.go:117] "RemoveContainer" containerID="45f459d7340dae6e14a3cdcabc82321380f9a9e9f5438f441e1bbc2fb5959b44" Jul 23 00:12:05.243006 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.242983 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm" event={"ID":"73b367b0-9408-4fcd-9535-cb198cfd7e34","Type":"ContainerStarted","Data":"93ded81b5fb7cab7b3c4c57173ede6579e6c6d70faf2b8f0b793ab6678131b98"} Jul 23 00:12:05.273450 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.273378 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm" podStartSLOduration=1.566663652 podStartE2EDuration="4.273357273s" totalImagesPullingTime="2.706693621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:05.271867294 +0000 UTC m=+245.232605924" watchObservedRunningTime="2026-07-23 00:12:05.273357273 +0000 UTC m=+245.234095896" Jul 23 00:12:05.594640 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.594547 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:05.594796 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.594646 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:05.594796 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:05.594680 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:12:05.594796 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:05.594758 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:12:05.594796 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:05.594761 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm podName:1c3cef2c-367d-4cea-bf38-e952f424ff69 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:06.594739605 +0000 UTC m=+246.555478203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9db4t" (UID: "1c3cef2c-367d-4cea-bf38-e952f424ff69") : configmap "kube-rbac-proxy" not found Jul 23 00:12:05.594932 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:05.594809 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls podName:1c3cef2c-367d-4cea-bf38-e952f424ff69 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:06.594793292 +0000 UTC m=+246.555531915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9db4t" (UID: "1c3cef2c-367d-4cea-bf38-e952f424ff69") : secret "insights-runtime-extractor-tls" not found Jul 23 00:12:05.878442 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.878352 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:05.978986 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:05.978952 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:06.037193 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:06.037163 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:06.247582 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:06.247553 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:12:06.247984 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:06.247968 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/0.log" Jul 23 00:12:06.248047 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:06.248002 2555 generic.go:356] "Generic (PLEG): container finished" podID="19897756-2813-4980-865c-1bbb2012aa33" containerID="42f1dd93786bf8d8f839a85c5c2cbac2aec36c062a5b7d0c20d16cd26d7b2b1e" exitCode=255 Jul 23 00:12:06.248082 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:06.248031 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-98rlm" event={"ID":"19897756-2813-4980-865c-1bbb2012aa33","Type":"ContainerDied","Data":"42f1dd93786bf8d8f839a85c5c2cbac2aec36c062a5b7d0c20d16cd26d7b2b1e"} Jul 23 00:12:06.248131 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:06.248117 2555 scope.go:117] "RemoveContainer" containerID="45f459d7340dae6e14a3cdcabc82321380f9a9e9f5438f441e1bbc2fb5959b44" Jul 23 00:12:06.248331 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:06.248311 2555 scope.go:117] "RemoveContainer" containerID="42f1dd93786bf8d8f839a85c5c2cbac2aec36c062a5b7d0c20d16cd26d7b2b1e" Jul 23 00:12:06.248556 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:06.248527 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-98rlm_openshift-console-operator(19897756-2813-4980-865c-1bbb2012aa33)\"" pod="openshift-console-operator/console-operator-565b7b569d-98rlm" podUID="19897756-2813-4980-865c-1bbb2012aa33" Jul 23 00:12:06.419937 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:06.419900 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:06.521311 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:06.521227 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:06.602794 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:06.602763 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:06.602921 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:06.602818 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:06.602921 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:06.602906 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:12:06.603032 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:06.602932 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:12:06.603032 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:06.602967 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm podName:1c3cef2c-367d-4cea-bf38-e952f424ff69 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:08.60295231 +0000 UTC m=+248.563690906 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9db4t" (UID: "1c3cef2c-367d-4cea-bf38-e952f424ff69") : configmap "kube-rbac-proxy" not found Jul 23 00:12:06.603032 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:06.603012 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls podName:1c3cef2c-367d-4cea-bf38-e952f424ff69 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:08.602989221 +0000 UTC m=+248.563727841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9db4t" (UID: "1c3cef2c-367d-4cea-bf38-e952f424ff69") : secret "insights-runtime-extractor-tls" not found Jul 23 00:12:06.911258 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:06.911232 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:06.961165 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:06.961133 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:07.050268 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:07.050239 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t75zr_225d2d8c-3a75-47f8-9943-2f67e3c62e8b/dns-node-resolver/0.log" Jul 23 00:12:07.121443 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:07.121379 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:07.223227 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:07.223150 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:07.251879 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:07.251853 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:12:07.252166 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:07.252153 2555 scope.go:117] "RemoveContainer" containerID="42f1dd93786bf8d8f839a85c5c2cbac2aec36c062a5b7d0c20d16cd26d7b2b1e" Jul 23 00:12:07.252320 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:07.252302 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-98rlm_openshift-console-operator(19897756-2813-4980-865c-1bbb2012aa33)\"" pod="openshift-console-operator/console-operator-565b7b569d-98rlm" podUID="19897756-2813-4980-865c-1bbb2012aa33" Jul 23 00:12:08.050791 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.050763 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pn5jp_754c2d81-3f3e-44d4-8589-fec36d904b15/node-ca/0.log" Jul 23 00:12:08.450539 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.450506 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f5wg8_5e4b524a-e6ce-4247-bf26-81fda530d64d/serve-healthcheck-canary/0.log" Jul 23 00:12:08.616439 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.616378 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:08.616632 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.616485 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:08.616632 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:08.616493 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:12:08.616632 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:08.616565 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm podName:1c3cef2c-367d-4cea-bf38-e952f424ff69 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:12.616544933 +0000 UTC m=+252.577283530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9db4t" (UID: "1c3cef2c-367d-4cea-bf38-e952f424ff69") : configmap "kube-rbac-proxy" not found Jul 23 00:12:08.616632 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:08.616575 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:12:08.616632 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:08.616613 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls podName:1c3cef2c-367d-4cea-bf38-e952f424ff69 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:12.616602794 +0000 UTC m=+252.577341390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9db4t" (UID: "1c3cef2c-367d-4cea-bf38-e952f424ff69") : secret "insights-runtime-extractor-tls" not found Jul 23 00:12:08.715348 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.715276 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-ftjtz"] Jul 23 00:12:08.719448 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.719433 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-ftjtz" Jul 23 00:12:08.722187 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.722164 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 23 00:12:08.723040 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.723025 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ll96l\"" Jul 23 00:12:08.723091 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.723056 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 23 00:12:08.723137 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.723124 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 23 00:12:08.723186 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.723133 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 23 00:12:08.725798 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.725778 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-ftjtz"] Jul 23 00:12:08.754108 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.754076 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:08.818977 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.818938 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6fe3c7ce-1b54-4c0b-848a-31d11abd68d4-signing-cabundle\") pod \"service-ca-58598695f7-ftjtz\" (UID: \"6fe3c7ce-1b54-4c0b-848a-31d11abd68d4\") " pod="openshift-service-ca/service-ca-58598695f7-ftjtz" Jul 23 00:12:08.819138 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.819020 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6fe3c7ce-1b54-4c0b-848a-31d11abd68d4-signing-key\") pod \"service-ca-58598695f7-ftjtz\" (UID: \"6fe3c7ce-1b54-4c0b-848a-31d11abd68d4\") " pod="openshift-service-ca/service-ca-58598695f7-ftjtz" Jul 23 00:12:08.819138 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.819039 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnds8\" (UniqueName: \"kubernetes.io/projected/6fe3c7ce-1b54-4c0b-848a-31d11abd68d4-kube-api-access-pnds8\") pod \"service-ca-58598695f7-ftjtz\" (UID: \"6fe3c7ce-1b54-4c0b-848a-31d11abd68d4\") " pod="openshift-service-ca/service-ca-58598695f7-ftjtz" Jul 23 00:12:08.854356 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.854326 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:08.919968 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.919932 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6fe3c7ce-1b54-4c0b-848a-31d11abd68d4-signing-key\") pod \"service-ca-58598695f7-ftjtz\" (UID: \"6fe3c7ce-1b54-4c0b-848a-31d11abd68d4\") " pod="openshift-service-ca/service-ca-58598695f7-ftjtz" Jul 23 00:12:08.919968 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.919966 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnds8\" (UniqueName: \"kubernetes.io/projected/6fe3c7ce-1b54-4c0b-848a-31d11abd68d4-kube-api-access-pnds8\") pod \"service-ca-58598695f7-ftjtz\" (UID: \"6fe3c7ce-1b54-4c0b-848a-31d11abd68d4\") " pod="openshift-service-ca/service-ca-58598695f7-ftjtz" Jul 23 00:12:08.920166 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.920047 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6fe3c7ce-1b54-4c0b-848a-31d11abd68d4-signing-cabundle\") pod \"service-ca-58598695f7-ftjtz\" (UID: \"6fe3c7ce-1b54-4c0b-848a-31d11abd68d4\") " pod="openshift-service-ca/service-ca-58598695f7-ftjtz" Jul 23 00:12:08.920773 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.920748 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6fe3c7ce-1b54-4c0b-848a-31d11abd68d4-signing-cabundle\") pod \"service-ca-58598695f7-ftjtz\" (UID: \"6fe3c7ce-1b54-4c0b-848a-31d11abd68d4\") " pod="openshift-service-ca/service-ca-58598695f7-ftjtz" Jul 23 00:12:08.922336 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.922319 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6fe3c7ce-1b54-4c0b-848a-31d11abd68d4-signing-key\") pod \"service-ca-58598695f7-ftjtz\" (UID: \"6fe3c7ce-1b54-4c0b-848a-31d11abd68d4\") " pod="openshift-service-ca/service-ca-58598695f7-ftjtz" Jul 23 00:12:08.933401 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:08.933366 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnds8\" (UniqueName: \"kubernetes.io/projected/6fe3c7ce-1b54-4c0b-848a-31d11abd68d4-kube-api-access-pnds8\") pod \"service-ca-58598695f7-ftjtz\" (UID: \"6fe3c7ce-1b54-4c0b-848a-31d11abd68d4\") " pod="openshift-service-ca/service-ca-58598695f7-ftjtz" Jul 23 00:12:09.028846 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:09.028748 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-ftjtz" Jul 23 00:12:09.147834 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:09.147812 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-ftjtz"] Jul 23 00:12:09.150198 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:12:09.150164 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe3c7ce_1b54_4c0b_848a_31d11abd68d4.slice/crio-27dbc0ea8c6cad2e8a77a356054b137498c541a23171d974e3dffa60f6bb46cb WatchSource:0}: Error finding container 27dbc0ea8c6cad2e8a77a356054b137498c541a23171d974e3dffa60f6bb46cb: Status 404 returned error can't find the container with id 27dbc0ea8c6cad2e8a77a356054b137498c541a23171d974e3dffa60f6bb46cb Jul 23 00:12:09.222876 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:09.222833 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qm5ml\" (UID: \"165778da-7482-49cf-948f-5440431f12e1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" Jul 23 00:12:09.223035 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:09.222980 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 00:12:09.223095 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:09.223047 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls podName:165778da-7482-49cf-948f-5440431f12e1 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:17.223029191 +0000 UTC m=+257.183767788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-qm5ml" (UID: "165778da-7482-49cf-948f-5440431f12e1") : secret "cluster-monitoring-operator-tls" not found Jul 23 00:12:09.257939 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:09.257906 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-ftjtz" event={"ID":"6fe3c7ce-1b54-4c0b-848a-31d11abd68d4","Type":"ContainerStarted","Data":"27dbc0ea8c6cad2e8a77a356054b137498c541a23171d974e3dffa60f6bb46cb"} Jul 23 00:12:09.794726 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:09.794685 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:11.758034 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:11.757996 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:11.758535 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:11.758055 2555 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:11.758535 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:11.758412 2555 scope.go:117] "RemoveContainer" containerID="42f1dd93786bf8d8f839a85c5c2cbac2aec36c062a5b7d0c20d16cd26d7b2b1e" Jul 23 00:12:11.758638 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:11.758556 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-98rlm_openshift-console-operator(19897756-2813-4980-865c-1bbb2012aa33)\"" pod="openshift-console-operator/console-operator-565b7b569d-98rlm" podUID="19897756-2813-4980-865c-1bbb2012aa33" Jul 23 00:12:12.267535 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:12.267497 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-ftjtz" event={"ID":"6fe3c7ce-1b54-4c0b-848a-31d11abd68d4","Type":"ContainerStarted","Data":"7ea284fc4506946593af3b0dec9f692c8994891005f2deaef3b10281e564f5d2"} Jul 23 00:12:12.284959 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:12.284910 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-ftjtz" podStartSLOduration=2.150280353 podStartE2EDuration="4.284892498s" totalImagesPullingTime="2.134612145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:12.284131622 +0000 UTC m=+252.244870242" watchObservedRunningTime="2026-07-23 00:12:12.284892498 +0000 UTC m=+252.245631116" Jul 23 00:12:12.350516 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:12.350474 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs\") pod \"network-metrics-daemon-czq2t\" (UID: \"907325cd-64b7-4241-8969-a32a19bab628\") " pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:12:12.350687 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:12.350652 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 23 00:12:12.350763 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:12.350749 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs podName:907325cd-64b7-4241-8969-a32a19bab628 nodeName:}" failed. No retries permitted until 2026-07-23 00:14:14.350726639 +0000 UTC m=+374.311465259 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs") pod "network-metrics-daemon-czq2t" (UID: "907325cd-64b7-4241-8969-a32a19bab628") : secret "metrics-daemon-secret" not found Jul 23 00:12:12.517132 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:12.517097 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:12.615661 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:12.615539 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:12.652746 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:12.652704 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:12.652923 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:12.652833 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:12.652923 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:12.652838 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:12:12.652923 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:12.652922 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm podName:1c3cef2c-367d-4cea-bf38-e952f424ff69 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:20.652900378 +0000 UTC m=+260.613638989 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9db4t" (UID: "1c3cef2c-367d-4cea-bf38-e952f424ff69") : configmap "kube-rbac-proxy" not found Jul 23 00:12:12.653093 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:12.653040 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:12:12.653149 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:12.653117 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls podName:1c3cef2c-367d-4cea-bf38-e952f424ff69 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:20.653100011 +0000 UTC m=+260.613838612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9db4t" (UID: "1c3cef2c-367d-4cea-bf38-e952f424ff69") : secret "insights-runtime-extractor-tls" not found Jul 23 00:12:12.718460 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:12.718420 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:12.817733 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:12.817643 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:12.918303 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:12.918261 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:13.020807 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:13.020764 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:13.452799 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:13.452760 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:13.837831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:13.837732 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:14.243540 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:14.243505 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:15.036131 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:15.036091 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:15.244557 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:15.244501 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:15.450021 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:15.449995 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:15.635851 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:15.635820 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:15.845891 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:15.845803 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:15.902790 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:15.902758 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:16.234973 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:16.234942 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:16.554105 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:16.554022 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:16.699405 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:16.699361 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:16.802723 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:16.802690 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:16.900163 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:16.900078 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:17.236988 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:17.236943 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:17.290814 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:17.290781 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qm5ml\" (UID: \"165778da-7482-49cf-948f-5440431f12e1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" Jul 23 00:12:17.290975 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:17.290945 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 00:12:17.291034 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:17.291027 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls podName:165778da-7482-49cf-948f-5440431f12e1 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:33.291004541 +0000 UTC m=+273.251743158 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-qm5ml" (UID: "165778da-7482-49cf-948f-5440431f12e1") : secret "cluster-monitoring-operator-tls" not found Jul 23 00:12:17.641131 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:17.641052 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:17.840119 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:17.840082 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:18.113944 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:18.113913 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:18.443427 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:18.443385 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:18.842977 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:18.842897 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:18.901633 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:18.901604 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:19.234669 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:19.234643 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:19.441613 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:19.441582 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:19.848171 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:19.848139 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:20.638168 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:20.638136 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:20.717715 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:20.717677 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:20.717916 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:20.717744 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:20.717916 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:20.717835 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:12:20.717916 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:20.717903 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls podName:1c3cef2c-367d-4cea-bf38-e952f424ff69 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:36.71788518 +0000 UTC m=+276.678623777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9db4t" (UID: "1c3cef2c-367d-4cea-bf38-e952f424ff69") : secret "insights-runtime-extractor-tls" not found Jul 23 00:12:20.718327 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:20.718306 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c3cef2c-367d-4cea-bf38-e952f424ff69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:20.842012 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:20.841976 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:20.993415 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:20.993375 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:21.094614 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:21.094579 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:21.235141 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:21.235097 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:21.443652 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:21.443622 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:22.034601 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:22.034524 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:22.497291 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:22.497269 2555 scope.go:117] "RemoveContainer" containerID="42f1dd93786bf8d8f839a85c5c2cbac2aec36c062a5b7d0c20d16cd26d7b2b1e" Jul 23 00:12:22.640332 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:22.640299 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:22.856454 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:22.856366 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:23.042999 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:23.042964 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:23.241044 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:23.241011 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:23.299922 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:23.299896 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:12:23.300079 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:23.299971 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-98rlm" event={"ID":"19897756-2813-4980-865c-1bbb2012aa33","Type":"ContainerStarted","Data":"9772b59fed7953d1ae5586e5abd09e6fdde5b2d2707959898027e727f0bc5987"} Jul 23 00:12:23.300235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:23.300218 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:23.317426 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:23.317371 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-98rlm" podStartSLOduration=19.52027893 podStartE2EDuration="22.317359773s" totalImagesPullingTime="2.797080843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:23.316369934 +0000 UTC m=+263.277108582" watchObservedRunningTime="2026-07-23 00:12:23.317359773 +0000 UTC m=+263.278098392" Jul 23 00:12:23.399089 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:23.399062 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-98rlm" Jul 23 00:12:23.572938 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:23.572858 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:23.677559 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:23.677525 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:23.870427 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:23.870332 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:24.256429 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:24.256375 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:24.312357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:24.312327 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:24.662008 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:24.661979 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:24.879830 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:24.879802 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:24.981019 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:24.980935 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:25.262978 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:25.262894 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:25.362930 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:25.362898 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:25.547088 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:25.547004 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:26.065381 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:26.065348 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:27.880013 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:27.879979 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:27.936521 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:27.936491 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:29.609172 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:29.609142 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-8cg64"] Jul 23 00:12:29.613488 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:29.613468 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-8cg64" Jul 23 00:12:29.615998 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:29.615980 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-454bj\"" Jul 23 00:12:29.616103 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:29.615987 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 00:12:29.616269 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:29.616252 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 00:12:29.622534 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:29.622511 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-8cg64"] Jul 23 00:12:29.684625 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:29.684597 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqftb\" (UniqueName: \"kubernetes.io/projected/4faf92ae-a088-43b4-a83f-ab0cab855e27-kube-api-access-qqftb\") pod \"downloads-5bc67dcd46-8cg64\" (UID: \"4faf92ae-a088-43b4-a83f-ab0cab855e27\") " pod="openshift-console/downloads-5bc67dcd46-8cg64" Jul 23 00:12:29.696810 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:29.696771 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:29.785269 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:29.785242 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqftb\" (UniqueName: \"kubernetes.io/projected/4faf92ae-a088-43b4-a83f-ab0cab855e27-kube-api-access-qqftb\") pod \"downloads-5bc67dcd46-8cg64\" (UID: \"4faf92ae-a088-43b4-a83f-ab0cab855e27\") " pod="openshift-console/downloads-5bc67dcd46-8cg64" Jul 23 00:12:29.793182 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:29.793157 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqftb\" (UniqueName: \"kubernetes.io/projected/4faf92ae-a088-43b4-a83f-ab0cab855e27-kube-api-access-qqftb\") pod \"downloads-5bc67dcd46-8cg64\" (UID: \"4faf92ae-a088-43b4-a83f-ab0cab855e27\") " pod="openshift-console/downloads-5bc67dcd46-8cg64" Jul 23 00:12:29.921867 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:29.921840 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-8cg64" Jul 23 00:12:30.045914 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:30.045887 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-8cg64"] Jul 23 00:12:30.049051 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:12:30.049023 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4faf92ae_a088_43b4_a83f_ab0cab855e27.slice/crio-8bc0571a0133bd3015bf49c9b8eca382808ad79edc91c68641193dd0598abc1b WatchSource:0}: Error finding container 8bc0571a0133bd3015bf49c9b8eca382808ad79edc91c68641193dd0598abc1b: Status 404 returned error can't find the container with id 8bc0571a0133bd3015bf49c9b8eca382808ad79edc91c68641193dd0598abc1b Jul 23 00:12:30.326220 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:30.326126 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-8cg64" event={"ID":"4faf92ae-a088-43b4-a83f-ab0cab855e27","Type":"ContainerStarted","Data":"8bc0571a0133bd3015bf49c9b8eca382808ad79edc91c68641193dd0598abc1b"} Jul 23 00:12:31.255052 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:31.255003 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:33.314668 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:33.314612 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qm5ml\" (UID: \"165778da-7482-49cf-948f-5440431f12e1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" Jul 23 00:12:33.317721 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:33.317690 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/165778da-7482-49cf-948f-5440431f12e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-qm5ml\" (UID: \"165778da-7482-49cf-948f-5440431f12e1\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" Jul 23 00:12:33.562210 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:33.562176 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" Jul 23 00:12:33.701147 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:33.701114 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml"] Jul 23 00:12:33.704294 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:12:33.704258 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165778da_7482_49cf_948f_5440431f12e1.slice/crio-48dcfc27b4c899ee126636fb6a4cbbe8e454153b4f20c3a273063183d94b2d3d WatchSource:0}: Error finding container 48dcfc27b4c899ee126636fb6a4cbbe8e454153b4f20c3a273063183d94b2d3d: Status 404 returned error can't find the container with id 48dcfc27b4c899ee126636fb6a4cbbe8e454153b4f20c3a273063183d94b2d3d Jul 23 00:12:33.969933 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:33.969882 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:34.111643 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:34.111584 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:34.340005 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:34.339920 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" event={"ID":"165778da-7482-49cf-948f-5440431f12e1","Type":"ContainerStarted","Data":"48dcfc27b4c899ee126636fb6a4cbbe8e454153b4f20c3a273063183d94b2d3d"} Jul 23 00:12:35.912676 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:35.912645 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:36.015368 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:36.015339 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:36.074624 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:36.074587 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg5wc"] Jul 23 00:12:36.077881 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:36.077855 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg5wc" Jul 23 00:12:36.080284 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:36.080226 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-x99t6\"" Jul 23 00:12:36.080449 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:36.080309 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 23 00:12:36.087131 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:36.086915 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg5wc"] Jul 23 00:12:36.091130 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:36.091019 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:36.139813 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:36.139776 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ef62896d-92ba-4312-bbf5-dcb3b85131eb-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-sg5wc\" (UID: \"ef62896d-92ba-4312-bbf5-dcb3b85131eb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg5wc" Jul 23 00:12:36.241137 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:36.241054 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ef62896d-92ba-4312-bbf5-dcb3b85131eb-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-sg5wc\" (UID: \"ef62896d-92ba-4312-bbf5-dcb3b85131eb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg5wc" Jul 23 00:12:36.241282 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:36.241187 2555 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 23 00:12:36.241282 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:36.241244 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef62896d-92ba-4312-bbf5-dcb3b85131eb-tls-certificates podName:ef62896d-92ba-4312-bbf5-dcb3b85131eb nodeName:}" failed. No retries permitted until 2026-07-23 00:12:36.741227085 +0000 UTC m=+276.701965701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/ef62896d-92ba-4312-bbf5-dcb3b85131eb-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-sg5wc" (UID: "ef62896d-92ba-4312-bbf5-dcb3b85131eb") : secret "prometheus-operator-admission-webhook-tls" not found Jul 23 00:12:36.348049 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:36.348013 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" event={"ID":"165778da-7482-49cf-948f-5440431f12e1","Type":"ContainerStarted","Data":"6797a6d585c585c5e15493774856a482ee222990caf40b5e426fdd8ccecc8f4f"} Jul 23 00:12:36.363874 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:36.363817 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-qm5ml" podStartSLOduration=33.497553357 podStartE2EDuration="35.363797139s" totalImagesPullingTime="1.866243782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:36.363238405 +0000 UTC m=+276.323977025" watchObservedRunningTime="2026-07-23 00:12:36.363797139 +0000 UTC m=+276.324535760" Jul 23 00:12:36.744993 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:36.744965 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:36.745172 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:36.745032 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ef62896d-92ba-4312-bbf5-dcb3b85131eb-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-sg5wc\" (UID: \"ef62896d-92ba-4312-bbf5-dcb3b85131eb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg5wc" Jul 23 00:12:36.747617 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:36.747595 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c3cef2c-367d-4cea-bf38-e952f424ff69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9db4t\" (UID: \"1c3cef2c-367d-4cea-bf38-e952f424ff69\") " pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:36.747617 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:36.747604 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ef62896d-92ba-4312-bbf5-dcb3b85131eb-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-sg5wc\" (UID: \"ef62896d-92ba-4312-bbf5-dcb3b85131eb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg5wc" Jul 23 00:12:36.761685 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:36.761662 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:36.989561 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:36.989529 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg5wc" Jul 23 00:12:37.015716 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:37.015686 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9db4t" Jul 23 00:12:37.139165 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:37.139099 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg5wc"] Jul 23 00:12:37.141544 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:12:37.141510 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef62896d_92ba_4312_bbf5_dcb3b85131eb.slice/crio-93ac4e144d02326ce981c002983799a46159a6e5717d2966acf6e2588e162986 WatchSource:0}: Error finding container 93ac4e144d02326ce981c002983799a46159a6e5717d2966acf6e2588e162986: Status 404 returned error can't find the container with id 93ac4e144d02326ce981c002983799a46159a6e5717d2966acf6e2588e162986 Jul 23 00:12:37.173927 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:37.173902 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9db4t"] Jul 23 00:12:37.176320 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:12:37.176294 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3cef2c_367d_4cea_bf38_e952f424ff69.slice/crio-d435d4198ebda9b93af4facbbfed68521869d277636e62ea67a5605120520a7b WatchSource:0}: Error finding container d435d4198ebda9b93af4facbbfed68521869d277636e62ea67a5605120520a7b: Status 404 returned error can't find the container with id d435d4198ebda9b93af4facbbfed68521869d277636e62ea67a5605120520a7b Jul 23 00:12:37.353081 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:37.353001 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9db4t" event={"ID":"1c3cef2c-367d-4cea-bf38-e952f424ff69","Type":"ContainerStarted","Data":"d435d4198ebda9b93af4facbbfed68521869d277636e62ea67a5605120520a7b"} Jul 23 00:12:37.354250 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:37.354220 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg5wc" event={"ID":"ef62896d-92ba-4312-bbf5-dcb3b85131eb","Type":"ContainerStarted","Data":"93ac4e144d02326ce981c002983799a46159a6e5717d2966acf6e2588e162986"} Jul 23 00:12:39.364203 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.364163 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg5wc" event={"ID":"ef62896d-92ba-4312-bbf5-dcb3b85131eb","Type":"ContainerStarted","Data":"6d8702a96bd8353f9337a0ff2514fec7afec0b6a11bee9e303af5fa6a947dd6a"} Jul 23 00:12:39.364668 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.364379 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg5wc" Jul 23 00:12:39.369805 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.369785 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg5wc" Jul 23 00:12:39.379246 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.379209 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg5wc" podStartSLOduration=1.921553824 podStartE2EDuration="3.379197293s" totalImagesPullingTime="1.457643469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:12:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:39.378632686 +0000 UTC m=+279.339371305" watchObservedRunningTime="2026-07-23 00:12:39.379197293 +0000 UTC m=+279.339935890" Jul 23 00:12:39.899220 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.899186 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86c48b6cd6-r288z"] Jul 23 00:12:39.912483 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.912448 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86c48b6cd6-r288z"] Jul 23 00:12:39.912653 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.912621 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:39.915080 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.915057 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 00:12:39.915214 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.915138 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hkc92\"" Jul 23 00:12:39.915214 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.915060 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 00:12:39.916035 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.915989 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 00:12:39.916167 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.916068 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 00:12:39.916351 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.916335 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 00:12:39.976264 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.975815 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-oauth-serving-cert\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:39.976264 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.975869 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-oauth-config\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:39.976264 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.975914 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-serving-cert\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:39.976264 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.975989 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-config\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:39.976264 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.976102 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-service-ca\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:39.976264 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:39.976165 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4z82\" (UniqueName: \"kubernetes.io/projected/950d50c5-3425-4b12-a39b-7d2adc7278fa-kube-api-access-v4z82\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:40.077061 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.077017 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4z82\" (UniqueName: \"kubernetes.io/projected/950d50c5-3425-4b12-a39b-7d2adc7278fa-kube-api-access-v4z82\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:40.077273 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.077074 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-oauth-serving-cert\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:40.077273 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.077110 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-oauth-config\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:40.077273 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.077133 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-serving-cert\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:40.077273 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.077197 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-config\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:40.077273 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.077230 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-service-ca\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:40.077933 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.077882 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-config\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:40.077933 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.077889 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-oauth-serving-cert\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:40.078089 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.077933 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-service-ca\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:40.080333 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.080306 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-oauth-config\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:40.080530 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.080507 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-serving-cert\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:40.088150 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.088109 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4z82\" (UniqueName: \"kubernetes.io/projected/950d50c5-3425-4b12-a39b-7d2adc7278fa-kube-api-access-v4z82\") pod \"console-86c48b6cd6-r288z\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:40.098139 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.098113 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:40.134775 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.134741 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc"] Jul 23 00:12:40.142073 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.142046 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" Jul 23 00:12:40.146015 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.145439 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 00:12:40.146015 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.145548 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 23 00:12:40.146015 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.145651 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-r4x69\"" Jul 23 00:12:40.146015 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.145846 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 23 00:12:40.146844 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.146821 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc"] Jul 23 00:12:40.164435 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.164363 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:40.225098 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.224907 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:12:40.279875 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.279832 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/457d585b-f5fe-4bd7-a933-c958d30917ab-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-jfqbc\" (UID: \"457d585b-f5fe-4bd7-a933-c958d30917ab\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" Jul 23 00:12:40.280039 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.279911 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/457d585b-f5fe-4bd7-a933-c958d30917ab-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-jfqbc\" (UID: \"457d585b-f5fe-4bd7-a933-c958d30917ab\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" Jul 23 00:12:40.280039 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.279998 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/457d585b-f5fe-4bd7-a933-c958d30917ab-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-jfqbc\" (UID: \"457d585b-f5fe-4bd7-a933-c958d30917ab\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" Jul 23 00:12:40.280039 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.280024 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2klk\" (UniqueName: \"kubernetes.io/projected/457d585b-f5fe-4bd7-a933-c958d30917ab-kube-api-access-s2klk\") pod \"prometheus-operator-5945b5cfd-jfqbc\" (UID: \"457d585b-f5fe-4bd7-a933-c958d30917ab\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" Jul 23 00:12:40.381215 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.381112 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/457d585b-f5fe-4bd7-a933-c958d30917ab-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-jfqbc\" (UID: \"457d585b-f5fe-4bd7-a933-c958d30917ab\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" Jul 23 00:12:40.381215 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.381196 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/457d585b-f5fe-4bd7-a933-c958d30917ab-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-jfqbc\" (UID: \"457d585b-f5fe-4bd7-a933-c958d30917ab\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" Jul 23 00:12:40.381821 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:40.381277 2555 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 23 00:12:40.381821 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:40.381344 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/457d585b-f5fe-4bd7-a933-c958d30917ab-prometheus-operator-tls podName:457d585b-f5fe-4bd7-a933-c958d30917ab nodeName:}" failed. No retries permitted until 2026-07-23 00:12:40.881324804 +0000 UTC m=+280.842063401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/457d585b-f5fe-4bd7-a933-c958d30917ab-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-jfqbc" (UID: "457d585b-f5fe-4bd7-a933-c958d30917ab") : secret "prometheus-operator-tls" not found Jul 23 00:12:40.381821 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.381279 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/457d585b-f5fe-4bd7-a933-c958d30917ab-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-jfqbc\" (UID: \"457d585b-f5fe-4bd7-a933-c958d30917ab\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" Jul 23 00:12:40.381821 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.381623 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2klk\" (UniqueName: \"kubernetes.io/projected/457d585b-f5fe-4bd7-a933-c958d30917ab-kube-api-access-s2klk\") pod \"prometheus-operator-5945b5cfd-jfqbc\" (UID: \"457d585b-f5fe-4bd7-a933-c958d30917ab\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" Jul 23 00:12:40.382031 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.381983 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/457d585b-f5fe-4bd7-a933-c958d30917ab-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-jfqbc\" (UID: \"457d585b-f5fe-4bd7-a933-c958d30917ab\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" Jul 23 00:12:40.384405 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.384335 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/457d585b-f5fe-4bd7-a933-c958d30917ab-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-jfqbc\" (UID: \"457d585b-f5fe-4bd7-a933-c958d30917ab\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" Jul 23 00:12:40.392838 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.392789 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2klk\" (UniqueName: \"kubernetes.io/projected/457d585b-f5fe-4bd7-a933-c958d30917ab-kube-api-access-s2klk\") pod \"prometheus-operator-5945b5cfd-jfqbc\" (UID: \"457d585b-f5fe-4bd7-a933-c958d30917ab\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" Jul 23 00:12:40.434048 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.433994 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:40.886814 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.886728 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/457d585b-f5fe-4bd7-a933-c958d30917ab-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-jfqbc\" (UID: \"457d585b-f5fe-4bd7-a933-c958d30917ab\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" Jul 23 00:12:40.889684 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:40.889655 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/457d585b-f5fe-4bd7-a933-c958d30917ab-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-jfqbc\" (UID: \"457d585b-f5fe-4bd7-a933-c958d30917ab\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" Jul 23 00:12:41.054697 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:41.054655 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" Jul 23 00:12:42.043428 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:42.043365 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-qwwgq" podUID="359cf272-977d-4bcd-8e31-d2d45a8b440f" Jul 23 00:12:42.043428 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:42.043383 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5db59d744-b57zf" podUID="0a31e563-70c6-441d-bd67-cc3dadb8b3eb" Jul 23 00:12:42.376033 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:42.375955 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:12:42.376199 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:42.375961 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qwwgq" Jul 23 00:12:43.545270 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:43.545228 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:43.646023 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:43.645973 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:45.887344 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:45.887310 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f956d7d7b-64zg8"] Jul 23 00:12:45.891769 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:45.891750 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:45.900740 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:45.900051 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 23 00:12:45.901199 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:45.901176 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f956d7d7b-64zg8"] Jul 23 00:12:45.927764 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:45.927735 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:12:45.927928 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:45.927778 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:12:45.930758 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:45.930733 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cf272-977d-4bcd-8e31-d2d45a8b440f-metrics-tls\") pod \"dns-default-qwwgq\" (UID: \"359cf272-977d-4bcd-8e31-d2d45a8b440f\") " pod="openshift-dns/dns-default-qwwgq" Jul 23 00:12:45.930857 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:45.930798 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls\") pod \"image-registry-5db59d744-b57zf\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:12:45.978823 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:45.978786 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fhhqg\"" Jul 23 00:12:45.978823 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:45.978794 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-t8qkr\"" Jul 23 00:12:45.986661 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:45.986620 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qwwgq" Jul 23 00:12:45.986814 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:45.986746 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:12:46.029159 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.029127 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-service-ca\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.029334 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.029166 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-oauth-serving-cert\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.029334 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.029194 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjqnj\" (UniqueName: \"kubernetes.io/projected/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-kube-api-access-rjqnj\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.029334 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.029282 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-config\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.029334 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.029308 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-trusted-ca-bundle\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.029549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.029381 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-oauth-config\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.029549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.029455 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-serving-cert\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.130539 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.130495 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-oauth-config\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.130739 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.130549 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-serving-cert\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.130739 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.130602 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-service-ca\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.130739 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.130628 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-oauth-serving-cert\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.130739 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.130652 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjqnj\" (UniqueName: \"kubernetes.io/projected/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-kube-api-access-rjqnj\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.130947 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.130828 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-config\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.130947 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.130872 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-trusted-ca-bundle\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.131578 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.131518 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-oauth-serving-cert\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.131578 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.131519 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-service-ca\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.131745 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.131581 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-config\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.131836 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.131807 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-trusted-ca-bundle\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.133290 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.133272 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-oauth-config\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.133520 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.133500 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-serving-cert\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.138681 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.138624 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjqnj\" (UniqueName: \"kubernetes.io/projected/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-kube-api-access-rjqnj\") pod \"console-5f956d7d7b-64zg8\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.203689 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.203633 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:12:46.246200 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.246164 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:46.342816 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.342783 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:46.400723 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:46.400633 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:47.288094 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:47.288061 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:47.388780 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:47.388750 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:49.691383 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:49.691298 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86c48b6cd6-r288z"] Jul 23 00:12:49.694890 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:12:49.694856 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950d50c5_3425_4b12_a39b_7d2adc7278fa.slice/crio-f2a60058c648d68abc79704ebe0b9829e2fbb3e65fe77aeb3fc41b6b89ec570e WatchSource:0}: Error finding container f2a60058c648d68abc79704ebe0b9829e2fbb3e65fe77aeb3fc41b6b89ec570e: Status 404 returned error can't find the container with id f2a60058c648d68abc79704ebe0b9829e2fbb3e65fe77aeb3fc41b6b89ec570e Jul 23 00:12:49.705846 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:49.705754 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f956d7d7b-64zg8"] Jul 23 00:12:49.709562 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:12:49.709449 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4f0e5f_b1d6_41a4_b135_22c806c131ee.slice/crio-84404ab107a651056dfb6298b9e666d92f66d278b3eb54be6db0d5a0b6e1b1ec WatchSource:0}: Error finding container 84404ab107a651056dfb6298b9e666d92f66d278b3eb54be6db0d5a0b6e1b1ec: Status 404 returned error can't find the container with id 84404ab107a651056dfb6298b9e666d92f66d278b3eb54be6db0d5a0b6e1b1ec Jul 23 00:12:49.716333 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:49.716250 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:49.754262 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:49.754175 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qwwgq"] Jul 23 00:12:49.757130 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:12:49.757107 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359cf272_977d_4bcd_8e31_d2d45a8b440f.slice/crio-3029c863be5a1c0654edeee63a10612ba22ed45a8612801f22c38f010c56ee09 WatchSource:0}: Error finding container 3029c863be5a1c0654edeee63a10612ba22ed45a8612801f22c38f010c56ee09: Status 404 returned error can't find the container with id 3029c863be5a1c0654edeee63a10612ba22ed45a8612801f22c38f010c56ee09 Jul 23 00:12:49.820776 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:49.820748 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:49.923824 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:49.923794 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc"] Jul 23 00:12:49.926960 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:12:49.926926 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457d585b_f5fe_4bd7_a933_c958d30917ab.slice/crio-776a2c7669ac27c1dcb243d2a7c2eaa0ca1d21b20cf158b4fb01268e3856137e WatchSource:0}: Error finding container 776a2c7669ac27c1dcb243d2a7c2eaa0ca1d21b20cf158b4fb01268e3856137e: Status 404 returned error can't find the container with id 776a2c7669ac27c1dcb243d2a7c2eaa0ca1d21b20cf158b4fb01268e3856137e Jul 23 00:12:49.929169 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:49.929080 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5db59d744-b57zf"] Jul 23 00:12:49.932079 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:12:49.932049 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a31e563_70c6_441d_bd67_cc3dadb8b3eb.slice/crio-93eeb941d698365ef4bac6613bb7140bbcd38cabfe6ca028ba93be3c3b93938a WatchSource:0}: Error finding container 93eeb941d698365ef4bac6613bb7140bbcd38cabfe6ca028ba93be3c3b93938a: Status 404 returned error can't find the container with id 93eeb941d698365ef4bac6613bb7140bbcd38cabfe6ca028ba93be3c3b93938a Jul 23 00:12:50.413699 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:50.412880 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5db59d744-b57zf" event={"ID":"0a31e563-70c6-441d-bd67-cc3dadb8b3eb","Type":"ContainerStarted","Data":"f6348e97cf3a209e4ad507b41abaec18ef906635546d77b6fcf6a8ee289e3b7c"} Jul 23 00:12:50.413699 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:50.412940 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5db59d744-b57zf" event={"ID":"0a31e563-70c6-441d-bd67-cc3dadb8b3eb","Type":"ContainerStarted","Data":"93eeb941d698365ef4bac6613bb7140bbcd38cabfe6ca028ba93be3c3b93938a"} Jul 23 00:12:50.413699 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:50.413466 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:12:50.420758 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:50.420725 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-8cg64" event={"ID":"4faf92ae-a088-43b4-a83f-ab0cab855e27","Type":"ContainerStarted","Data":"3a4604c1e11888dea000c182ec4b64b989a7c48120c612ba3201a034bab79643"} Jul 23 00:12:50.423306 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:50.423249 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-8cg64" Jul 23 00:12:50.424990 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:50.424956 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qwwgq" event={"ID":"359cf272-977d-4bcd-8e31-d2d45a8b440f","Type":"ContainerStarted","Data":"3029c863be5a1c0654edeee63a10612ba22ed45a8612801f22c38f010c56ee09"} Jul 23 00:12:50.425885 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:50.425847 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-8cg64" Jul 23 00:12:50.426531 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:50.426469 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" event={"ID":"457d585b-f5fe-4bd7-a933-c958d30917ab","Type":"ContainerStarted","Data":"776a2c7669ac27c1dcb243d2a7c2eaa0ca1d21b20cf158b4fb01268e3856137e"} Jul 23 00:12:50.430209 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:50.430143 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86c48b6cd6-r288z" event={"ID":"950d50c5-3425-4b12-a39b-7d2adc7278fa","Type":"ContainerStarted","Data":"f2a60058c648d68abc79704ebe0b9829e2fbb3e65fe77aeb3fc41b6b89ec570e"} Jul 23 00:12:50.432467 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:50.432418 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9db4t" event={"ID":"1c3cef2c-367d-4cea-bf38-e952f424ff69","Type":"ContainerStarted","Data":"49b1f961ad47506ad69fe67f4a67feeaa423eda6897ed4c7c8f877497f488bfc"} Jul 23 00:12:50.435873 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:50.435849 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f956d7d7b-64zg8" event={"ID":"dd4f0e5f-b1d6-41a4-b135-22c806c131ee","Type":"ContainerStarted","Data":"84404ab107a651056dfb6298b9e666d92f66d278b3eb54be6db0d5a0b6e1b1ec"} Jul 23 00:12:50.439987 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:50.439038 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5db59d744-b57zf" podStartSLOduration=303.439021321 podStartE2EDuration="5m3.439021321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:50.437906532 +0000 UTC m=+290.398645144" watchObservedRunningTime="2026-07-23 00:12:50.439021321 +0000 UTC m=+290.399759942" Jul 23 00:12:50.457801 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:50.457142 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-8cg64" podStartSLOduration=1.956724025 podStartE2EDuration="21.457122819s" totalImagesPullingTime="19.500398794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:50.456602451 +0000 UTC m=+290.417341071" watchObservedRunningTime="2026-07-23 00:12:50.457122819 +0000 UTC m=+290.417861439" Jul 23 00:12:51.834340 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:51.834161 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5db59d744-b57zf"] Jul 23 00:12:51.914582 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:51.914514 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:51.976926 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:51.976894 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:52.202241 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:52.202131 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:52.302153 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:52.302077 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:54.937620 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:54.937532 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:55.038134 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:55.038095 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:55.497840 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:12:55.497798 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-czq2t" podUID="907325cd-64b7-4241-8969-a32a19bab628" Jul 23 00:12:58.813046 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:58.812978 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:12:59.484695 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:59.484647 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qwwgq" event={"ID":"359cf272-977d-4bcd-8e31-d2d45a8b440f","Type":"ContainerStarted","Data":"0bc98796a17e780425e48e4dee8c88dd742630708f29d0b4ce1dae6b8046f7df"} Jul 23 00:12:59.484998 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:59.484728 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qwwgq" event={"ID":"359cf272-977d-4bcd-8e31-d2d45a8b440f","Type":"ContainerStarted","Data":"56d20de622146aa38b8de80dc620924178b20c2edd073c1437cea71699dccb49"} Jul 23 00:12:59.484998 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:59.484790 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qwwgq" Jul 23 00:12:59.486677 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:59.486630 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" event={"ID":"457d585b-f5fe-4bd7-a933-c958d30917ab","Type":"ContainerStarted","Data":"2a0cf65c7f5d32f04b94ba81a89ce91eae07f92a385b4c2a486915f1599ec7db"} Jul 23 00:12:59.486799 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:59.486688 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" event={"ID":"457d585b-f5fe-4bd7-a933-c958d30917ab","Type":"ContainerStarted","Data":"6444dbd58ddd171c299c6fef2bafbf74af4c7812f4ec3fd7adf56031955ca3db"} Jul 23 00:12:59.488287 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:59.488263 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86c48b6cd6-r288z" event={"ID":"950d50c5-3425-4b12-a39b-7d2adc7278fa","Type":"ContainerStarted","Data":"b4403783f85f320cd943475a4332d5aedaa9b7e01132b1d3a68f69e195ac7646"} Jul 23 00:12:59.490426 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:59.490384 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9db4t" event={"ID":"1c3cef2c-367d-4cea-bf38-e952f424ff69","Type":"ContainerStarted","Data":"7bb00ed7b9aa2bd3c91607b53937824a7e9bbdcbb4870793e388974e637cb5de"} Jul 23 00:12:59.492111 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:59.492085 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f956d7d7b-64zg8" event={"ID":"dd4f0e5f-b1d6-41a4-b135-22c806c131ee","Type":"ContainerStarted","Data":"8c881c1517c4ff983b04a09054716516c0ee5985e169b3db94a42aa84e8762d4"} Jul 23 00:12:59.503309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:59.503268 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qwwgq" podStartSLOduration=255.799878826 podStartE2EDuration="4m24.503255894s" totalImagesPullingTime="8.703377068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:59.502158393 +0000 UTC m=+299.462897013" watchObservedRunningTime="2026-07-23 00:12:59.503255894 +0000 UTC m=+299.463994513" Jul 23 00:12:59.520323 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:59.520276 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f956d7d7b-64zg8" podStartSLOduration=5.763547685 podStartE2EDuration="14.52026352s" totalImagesPullingTime="8.756715835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:59.519160102 +0000 UTC m=+299.479898724" watchObservedRunningTime="2026-07-23 00:12:59.52026352 +0000 UTC m=+299.481002139" Jul 23 00:12:59.537518 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:59.537478 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86c48b6cd6-r288z" podStartSLOduration=11.770869912 podStartE2EDuration="20.53746469s" totalImagesPullingTime="8.766594778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:59.53658018 +0000 UTC m=+299.497318799" watchObservedRunningTime="2026-07-23 00:12:59.53746469 +0000 UTC m=+299.498203309" Jul 23 00:12:59.553708 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:12:59.553544 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-jfqbc" podStartSLOduration=11.063850404 podStartE2EDuration="19.553505411s" totalImagesPullingTime="8.489655007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:59.553103309 +0000 UTC m=+299.513841928" watchObservedRunningTime="2026-07-23 00:12:59.553505411 +0000 UTC m=+299.514244011" Jul 23 00:13:00.225521 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:00.225466 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:13:00.225951 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:00.225545 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:13:00.230844 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:00.230820 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:13:00.358899 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:00.358764 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:13:00.360786 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:00.360757 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:13:00.372270 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:00.372246 2555 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 00:13:00.501587 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:00.501424 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:13:00.723215 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:00.627600 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:00.723215 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:00.681900 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:01.233888 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.233854 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:01.381211 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.381147 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:01.461245 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.461207 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:01.464909 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.464882 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4"] Jul 23 00:13:01.534100 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.534021 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:01.570766 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.570596 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4"] Jul 23 00:13:01.570766 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.570651 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-lmdpb"] Jul 23 00:13:01.570766 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.570710 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" Jul 23 00:13:01.573241 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.573180 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 23 00:13:01.573368 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.573246 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vwdrh\"" Jul 23 00:13:01.573470 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.573410 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 23 00:13:01.594783 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.594746 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8jgfv"] Jul 23 00:13:01.594936 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.594917 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.597789 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.597528 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 23 00:13:01.597789 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.597758 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 23 00:13:01.597789 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.597785 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7hqqq\"" Jul 23 00:13:01.598000 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.597875 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 23 00:13:01.612707 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.612676 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-lmdpb"] Jul 23 00:13:01.612825 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.612816 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.615217 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.615190 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 00:13:01.615217 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.615210 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 00:13:01.615385 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.615250 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 00:13:01.615633 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.615606 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rtrwh\"" Jul 23 00:13:01.637404 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.637369 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:01.678575 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.678538 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5csz\" (UniqueName: \"kubernetes.io/projected/111a2305-1c97-4e01-907e-eeef54a558bb-kube-api-access-b5csz\") pod \"openshift-state-metrics-79f6958cdf-d56b4\" (UID: \"111a2305-1c97-4e01-907e-eeef54a558bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" Jul 23 00:13:01.678768 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.678602 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/111a2305-1c97-4e01-907e-eeef54a558bb-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-d56b4\" (UID: \"111a2305-1c97-4e01-907e-eeef54a558bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" Jul 23 00:13:01.678768 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.678683 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/111a2305-1c97-4e01-907e-eeef54a558bb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-d56b4\" (UID: \"111a2305-1c97-4e01-907e-eeef54a558bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" Jul 23 00:13:01.678919 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.678895 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/111a2305-1c97-4e01-907e-eeef54a558bb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-d56b4\" (UID: \"111a2305-1c97-4e01-907e-eeef54a558bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" Jul 23 00:13:01.741819 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.741761 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:01.780016 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.779980 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8db7756f-5d48-404c-9529-4601b1fedbd0-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.780016 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780023 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/722842b0-0473-4e43-bbd2-fed93e985695-root\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.780555 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780106 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/111a2305-1c97-4e01-907e-eeef54a558bb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-d56b4\" (UID: \"111a2305-1c97-4e01-907e-eeef54a558bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" Jul 23 00:13:01.780555 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780136 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsnc9\" (UniqueName: \"kubernetes.io/projected/8db7756f-5d48-404c-9529-4601b1fedbd0-kube-api-access-qsnc9\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.780555 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780157 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8db7756f-5d48-404c-9529-4601b1fedbd0-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.780555 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780177 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8db7756f-5d48-404c-9529-4601b1fedbd0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.780555 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780197 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.780555 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780219 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5csz\" (UniqueName: \"kubernetes.io/projected/111a2305-1c97-4e01-907e-eeef54a558bb-kube-api-access-b5csz\") pod \"openshift-state-metrics-79f6958cdf-d56b4\" (UID: \"111a2305-1c97-4e01-907e-eeef54a558bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" Jul 23 00:13:01.780555 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780245 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/722842b0-0473-4e43-bbd2-fed93e985695-sys\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.780555 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780268 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-tls\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.780555 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780329 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/111a2305-1c97-4e01-907e-eeef54a558bb-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-d56b4\" (UID: \"111a2305-1c97-4e01-907e-eeef54a558bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" Jul 23 00:13:01.780555 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780437 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-textfile\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.781069 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780674 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-wtmp\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.781069 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780709 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/722842b0-0473-4e43-bbd2-fed93e985695-metrics-client-ca\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.781069 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780738 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc28t\" (UniqueName: \"kubernetes.io/projected/722842b0-0473-4e43-bbd2-fed93e985695-kube-api-access-rc28t\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.781069 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780786 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8db7756f-5d48-404c-9529-4601b1fedbd0-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.781069 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780814 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8db7756f-5d48-404c-9529-4601b1fedbd0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.781069 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780855 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-accelerators-collector-config\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.781069 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.780927 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/111a2305-1c97-4e01-907e-eeef54a558bb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-d56b4\" (UID: \"111a2305-1c97-4e01-907e-eeef54a558bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" Jul 23 00:13:01.781507 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.781071 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/111a2305-1c97-4e01-907e-eeef54a558bb-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-d56b4\" (UID: \"111a2305-1c97-4e01-907e-eeef54a558bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" Jul 23 00:13:01.783802 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.783761 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/111a2305-1c97-4e01-907e-eeef54a558bb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-d56b4\" (UID: \"111a2305-1c97-4e01-907e-eeef54a558bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" Jul 23 00:13:01.784229 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.784117 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/111a2305-1c97-4e01-907e-eeef54a558bb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-d56b4\" (UID: \"111a2305-1c97-4e01-907e-eeef54a558bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" Jul 23 00:13:01.788909 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.788851 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5csz\" (UniqueName: \"kubernetes.io/projected/111a2305-1c97-4e01-907e-eeef54a558bb-kube-api-access-b5csz\") pod \"openshift-state-metrics-79f6958cdf-d56b4\" (UID: \"111a2305-1c97-4e01-907e-eeef54a558bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" Jul 23 00:13:01.882330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.881685 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-wtmp\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.882330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.881730 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/722842b0-0473-4e43-bbd2-fed93e985695-metrics-client-ca\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.882330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.881771 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc28t\" (UniqueName: \"kubernetes.io/projected/722842b0-0473-4e43-bbd2-fed93e985695-kube-api-access-rc28t\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.882330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.881889 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-wtmp\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.882330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.881983 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8db7756f-5d48-404c-9529-4601b1fedbd0-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.882330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.882010 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" Jul 23 00:13:01.882330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.882021 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8db7756f-5d48-404c-9529-4601b1fedbd0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.882330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.882063 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-accelerators-collector-config\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.882330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.882103 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8db7756f-5d48-404c-9529-4601b1fedbd0-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.882330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.882129 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/722842b0-0473-4e43-bbd2-fed93e985695-root\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.882330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.882203 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsnc9\" (UniqueName: \"kubernetes.io/projected/8db7756f-5d48-404c-9529-4601b1fedbd0-kube-api-access-qsnc9\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.882330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.882241 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8db7756f-5d48-404c-9529-4601b1fedbd0-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.882330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.882269 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8db7756f-5d48-404c-9529-4601b1fedbd0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.882330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.882309 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.882330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.882346 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/722842b0-0473-4e43-bbd2-fed93e985695-sys\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.883235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.882374 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-tls\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.883235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.882415 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8db7756f-5d48-404c-9529-4601b1fedbd0-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.883235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.882434 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-textfile\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.883235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.882506 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/722842b0-0473-4e43-bbd2-fed93e985695-metrics-client-ca\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.883235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.882704 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-textfile\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.883235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.882770 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/722842b0-0473-4e43-bbd2-fed93e985695-sys\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.883235 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:13:01.882786 2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 23 00:13:01.883235 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:13:01.882846 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-tls podName:722842b0-0473-4e43-bbd2-fed93e985695 nodeName:}" failed. No retries permitted until 2026-07-23 00:13:02.382828571 +0000 UTC m=+302.343567167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-tls") pod "node-exporter-8jgfv" (UID: "722842b0-0473-4e43-bbd2-fed93e985695") : secret "node-exporter-tls" not found Jul 23 00:13:01.883235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.883087 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/722842b0-0473-4e43-bbd2-fed93e985695-root\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.883892 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.883532 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8db7756f-5d48-404c-9529-4601b1fedbd0-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.883892 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.883724 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-accelerators-collector-config\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.885020 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.884986 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8db7756f-5d48-404c-9529-4601b1fedbd0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.885403 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.885352 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8db7756f-5d48-404c-9529-4601b1fedbd0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.886257 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.886217 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8db7756f-5d48-404c-9529-4601b1fedbd0-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.886758 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.886733 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.890815 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.890691 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc28t\" (UniqueName: \"kubernetes.io/projected/722842b0-0473-4e43-bbd2-fed93e985695-kube-api-access-rc28t\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:01.892091 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.891772 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsnc9\" (UniqueName: \"kubernetes.io/projected/8db7756f-5d48-404c-9529-4601b1fedbd0-kube-api-access-qsnc9\") pod \"kube-state-metrics-56c5594674-lmdpb\" (UID: \"8db7756f-5d48-404c-9529-4601b1fedbd0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.906906 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.906882 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" Jul 23 00:13:01.972874 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:01.972696 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:02.073631 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:02.073072 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:02.172940 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:02.172903 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:02.386366 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:02.386270 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-tls\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:02.389163 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:02.389140 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/722842b0-0473-4e43-bbd2-fed93e985695-node-exporter-tls\") pod \"node-exporter-8jgfv\" (UID: \"722842b0-0473-4e43-bbd2-fed93e985695\") " pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:02.466920 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:02.466889 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:02.523365 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:02.523331 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8jgfv" Jul 23 00:13:02.544891 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:02.544860 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:02.638887 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:13:02.638860 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722842b0_0473_4e43_bbd2_fed93e985695.slice/crio-8539628f8831dbe168a9ec34395f5fa672fc8195bd403fc38ed2e064bce3afef WatchSource:0}: Error finding container 8539628f8831dbe168a9ec34395f5fa672fc8195bd403fc38ed2e064bce3afef: Status 404 returned error can't find the container with id 8539628f8831dbe168a9ec34395f5fa672fc8195bd403fc38ed2e064bce3afef Jul 23 00:13:02.641217 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:02.641196 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:13:02.795741 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:02.795698 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-lmdpb"] Jul 23 00:13:02.800935 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:13:02.800899 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db7756f_5d48_404c_9529_4601b1fedbd0.slice/crio-6fe2ad685007a751f32c2d8c2aa166aa4246b1c2cb5c135be3149903b5cb171b WatchSource:0}: Error finding container 6fe2ad685007a751f32c2d8c2aa166aa4246b1c2cb5c135be3149903b5cb171b: Status 404 returned error can't find the container with id 6fe2ad685007a751f32c2d8c2aa166aa4246b1c2cb5c135be3149903b5cb171b Jul 23 00:13:02.812143 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:02.812122 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4"] Jul 23 00:13:02.814353 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:13:02.814327 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111a2305_1c97_4e01_907e_eeef54a558bb.slice/crio-3be083271c49e4b0cca123a6acce0e9c9c203ede8284ffd7df4469f80bcd7c05 WatchSource:0}: Error finding container 3be083271c49e4b0cca123a6acce0e9c9c203ede8284ffd7df4469f80bcd7c05: Status 404 returned error can't find the container with id 3be083271c49e4b0cca123a6acce0e9c9c203ede8284ffd7df4469f80bcd7c05 Jul 23 00:13:02.898913 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:02.898803 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:03.001008 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:03.000968 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:03.520603 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:03.520563 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" event={"ID":"111a2305-1c97-4e01-907e-eeef54a558bb","Type":"ContainerStarted","Data":"8015e834541da410022b051424909761ed6d8e97b9f6a06a22b1f455cbce910e"} Jul 23 00:13:03.521154 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:03.520637 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" event={"ID":"111a2305-1c97-4e01-907e-eeef54a558bb","Type":"ContainerStarted","Data":"ad1ae7490810231228a9b24029a7dcb749252b1e77f1db8af689fe7f524a34de"} Jul 23 00:13:03.521154 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:03.520668 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" event={"ID":"111a2305-1c97-4e01-907e-eeef54a558bb","Type":"ContainerStarted","Data":"3be083271c49e4b0cca123a6acce0e9c9c203ede8284ffd7df4469f80bcd7c05"} Jul 23 00:13:03.521925 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:03.521892 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" event={"ID":"8db7756f-5d48-404c-9529-4601b1fedbd0","Type":"ContainerStarted","Data":"6fe2ad685007a751f32c2d8c2aa166aa4246b1c2cb5c135be3149903b5cb171b"} Jul 23 00:13:03.524265 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:03.524243 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9db4t" event={"ID":"1c3cef2c-367d-4cea-bf38-e952f424ff69","Type":"ContainerStarted","Data":"17be96ad2e30748649d6e6a9a4dca81ecb4d57aaebc56bcd1f7b0bb4b264473c"} Jul 23 00:13:03.525547 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:03.525511 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jgfv" event={"ID":"722842b0-0473-4e43-bbd2-fed93e985695","Type":"ContainerStarted","Data":"8539628f8831dbe168a9ec34395f5fa672fc8195bd403fc38ed2e064bce3afef"} Jul 23 00:13:03.541052 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:03.540992 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9db4t" podStartSLOduration=33.889154913 podStartE2EDuration="59.540975235s" totalImagesPullingTime="25.651820322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:12:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:13:03.540536931 +0000 UTC m=+303.501275551" watchObservedRunningTime="2026-07-23 00:13:03.540975235 +0000 UTC m=+303.501713853" Jul 23 00:13:03.542479 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:03.542442 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:04.462978 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:04.462945 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:04.550425 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:04.550367 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:04.643687 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:04.643646 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:05.068962 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:05.068931 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:05.169703 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:05.169624 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:05.269613 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:05.269582 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:05.996639 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:05.996598 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:06.096628 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.096538 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:06.204167 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.204132 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:13:06.204375 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.204234 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:13:06.209112 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.209086 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:13:06.306910 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.306873 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:06.330028 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.329996 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f956d7d7b-64zg8"] Jul 23 00:13:06.405950 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.405879 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:06.428980 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.428947 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b59545fd4-ckp28"] Jul 23 00:13:06.446949 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.446921 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.448879 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.448850 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b59545fd4-ckp28"] Jul 23 00:13:06.535650 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.535620 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:06.538975 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.538942 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" event={"ID":"111a2305-1c97-4e01-907e-eeef54a558bb","Type":"ContainerStarted","Data":"e4b699e1316e08349a91c48a063c9f3844dac505fefbd95da62d434e3febb31b"} Jul 23 00:13:06.540766 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.540743 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" event={"ID":"8db7756f-5d48-404c-9529-4601b1fedbd0","Type":"ContainerStarted","Data":"cc4a6c8bfdd61c364f2e44bcba562925577bb38afd803e3e4a8a5dce9c64f859"} Jul 23 00:13:06.540883 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.540813 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" event={"ID":"8db7756f-5d48-404c-9529-4601b1fedbd0","Type":"ContainerStarted","Data":"652c9e2ccac37492612cdf51fb423ccdc019cc1effa2685caaf0941ca1f0dacc"} Jul 23 00:13:06.540883 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.540846 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" event={"ID":"8db7756f-5d48-404c-9529-4601b1fedbd0","Type":"ContainerStarted","Data":"6b1b6594d0193cd0c4520b7e6019eea844f8eaf243271f23a52c798b1af71755"} Jul 23 00:13:06.542261 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.542242 2555 generic.go:356] "Generic (PLEG): container finished" podID="722842b0-0473-4e43-bbd2-fed93e985695" containerID="21691f5d5b20470fdfb28569f063787b9cca934a02974761dde0aa98d5024df3" exitCode=0 Jul 23 00:13:06.542356 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.542312 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jgfv" event={"ID":"722842b0-0473-4e43-bbd2-fed93e985695","Type":"ContainerDied","Data":"21691f5d5b20470fdfb28569f063787b9cca934a02974761dde0aa98d5024df3"} Jul 23 00:13:06.546305 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.546282 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:13:06.559150 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.559079 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-d56b4" podStartSLOduration=3.200008743 podStartE2EDuration="5.55906498s" totalImagesPullingTime="2.359056237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:13:06.558010193 +0000 UTC m=+306.518748811" watchObservedRunningTime="2026-07-23 00:13:06.55906498 +0000 UTC m=+306.519803601" Jul 23 00:13:06.579798 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.579758 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-lmdpb" podStartSLOduration=2.827082743 podStartE2EDuration="5.579742936s" totalImagesPullingTime="2.752660193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:13:06.578964048 +0000 UTC m=+306.539702668" watchObservedRunningTime="2026-07-23 00:13:06.579742936 +0000 UTC m=+306.540481560" Jul 23 00:13:06.624618 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.624593 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-trusted-ca-bundle\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.624734 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.624659 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-service-ca\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.625592 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.624907 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-oauth-serving-cert\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.625592 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.624963 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bcxl\" (UniqueName: \"kubernetes.io/projected/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-kube-api-access-4bcxl\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.625592 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.625041 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-config\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.625592 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.625083 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-serving-cert\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.625592 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.625180 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-oauth-config\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.726156 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.726132 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-oauth-serving-cert\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.726269 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.726165 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bcxl\" (UniqueName: \"kubernetes.io/projected/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-kube-api-access-4bcxl\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.726269 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.726187 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-config\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.726269 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.726205 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-serving-cert\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.726459 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.726302 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-oauth-config\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.726459 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.726408 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-trusted-ca-bundle\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.726459 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.726447 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-service-ca\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.727016 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.726991 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-oauth-serving-cert\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.727152 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.727091 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-config\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.727152 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.727101 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-service-ca\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.727594 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.727574 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-trusted-ca-bundle\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.728786 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.728763 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-oauth-config\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.728868 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.728823 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-serving-cert\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.738267 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.738240 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bcxl\" (UniqueName: \"kubernetes.io/projected/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-kube-api-access-4bcxl\") pod \"console-6b59545fd4-ckp28\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.758405 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.758364 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:06.914503 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:06.914473 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b59545fd4-ckp28"] Jul 23 00:13:06.917437 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:13:06.917408 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c1e6cf_5ef3_4f78_94a6_f564ae3af861.slice/crio-f03073df5f3eaf423497965a5a1ce5a6279dea07ac12e7e73af50bca2495dee5 WatchSource:0}: Error finding container f03073df5f3eaf423497965a5a1ce5a6279dea07ac12e7e73af50bca2495dee5: Status 404 returned error can't find the container with id f03073df5f3eaf423497965a5a1ce5a6279dea07ac12e7e73af50bca2495dee5 Jul 23 00:13:07.547759 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.547726 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b59545fd4-ckp28" event={"ID":"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861","Type":"ContainerStarted","Data":"3c7191808329331ffb9d9ff86ead62c3bb7195f819766f0a18b0425d975b55b4"} Jul 23 00:13:07.548215 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.547775 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b59545fd4-ckp28" event={"ID":"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861","Type":"ContainerStarted","Data":"f03073df5f3eaf423497965a5a1ce5a6279dea07ac12e7e73af50bca2495dee5"} Jul 23 00:13:07.549694 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.549663 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jgfv" event={"ID":"722842b0-0473-4e43-bbd2-fed93e985695","Type":"ContainerStarted","Data":"e886d92cde0d4fe09253ecfd5e33cd06d420a819bf9824e5671e494cb23d33e1"} Jul 23 00:13:07.549808 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.549717 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jgfv" event={"ID":"722842b0-0473-4e43-bbd2-fed93e985695","Type":"ContainerStarted","Data":"9c2d95879095cc75058b7b3a42f7f94691246fd8beccab8e25c097a1285cf09f"} Jul 23 00:13:07.564852 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.564802 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b59545fd4-ckp28" podStartSLOduration=1.564783263 podStartE2EDuration="1.564783263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:13:07.564054335 +0000 UTC m=+307.524792955" watchObservedRunningTime="2026-07-23 00:13:07.564783263 +0000 UTC m=+307.525521883" Jul 23 00:13:07.584252 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.584200 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8jgfv" podStartSLOduration=3.570939699 podStartE2EDuration="6.584183636s" totalImagesPullingTime="2.891363727s" totalInitContainerRuntime="121.88021ms" isStatefulPod=true podCreationTimestamp="2026-07-23 00:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:13:07.58292755 +0000 UTC m=+307.543666171" watchObservedRunningTime="2026-07-23 00:13:07.584183636 +0000 UTC m=+307.544922255" Jul 23 00:13:07.675588 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.675554 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:07.684664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.684631 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 00:13:07.706162 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.706135 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 00:13:07.706326 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.706268 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.709641 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.709003 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 23 00:13:07.709641 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.709050 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 23 00:13:07.709641 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.709125 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 23 00:13:07.709641 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.709249 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 23 00:13:07.709641 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.709341 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 23 00:13:07.709641 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.709350 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 23 00:13:07.709641 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.709467 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 23 00:13:07.710029 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.709799 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 23 00:13:07.710119 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.710084 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 23 00:13:07.710309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.710283 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 23 00:13:07.710516 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.710500 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 23 00:13:07.710613 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.710598 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-h6p4b\"" Jul 23 00:13:07.710706 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.710690 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 23 00:13:07.710895 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.710879 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 23 00:13:07.711025 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.711010 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9usec5k7h73hk\"" Jul 23 00:13:07.711214 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.711201 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 23 00:13:07.712995 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.712951 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 23 00:13:07.735497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.734613 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.735497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.734654 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.735497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.734735 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b953752-e60a-4a77-bc61-fe0e1e78a022-config-out\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.735497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.734758 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.735497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.734809 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.735497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.734838 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.735497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.734864 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.735497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.734889 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-web-config\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.735497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.734912 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.735497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.734941 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.735497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.734970 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.735497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.734998 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.735497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.735031 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.735497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.735054 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.735497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.735080 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.735497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.735110 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47nt\" (UniqueName: \"kubernetes.io/projected/1b953752-e60a-4a77-bc61-fe0e1e78a022-kube-api-access-x47nt\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.736405 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.735152 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.736405 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.735192 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.736405 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.735227 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-config\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.736405 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.735247 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b953752-e60a-4a77-bc61-fe0e1e78a022-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.835974 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.835896 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.835974 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.835936 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.835974 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.835955 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.836237 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.836116 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.836237 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.836161 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.836237 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.836193 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.836237 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.836227 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47nt\" (UniqueName: \"kubernetes.io/projected/1b953752-e60a-4a77-bc61-fe0e1e78a022-kube-api-access-x47nt\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.836360 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.836276 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.836360 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.836316 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.836360 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.836354 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-config\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.836505 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.836377 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b953752-e60a-4a77-bc61-fe0e1e78a022-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.836505 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.836450 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.836505 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.836473 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.836638 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.836517 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.836638 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.836530 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b953752-e60a-4a77-bc61-fe0e1e78a022-config-out\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.836638 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.836553 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.836638 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.836591 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.836829 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.836647 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.836829 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.836683 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.836829 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.836706 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-web-config\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.837967 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.837167 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.838596 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.838577 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.839595 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.839569 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.840760 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.839906 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.840760 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.840029 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.840760 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.840085 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.840760 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.840151 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.840760 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.840311 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.840760 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.840338 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.841111 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.840825 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.841111 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.840950 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.841224 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.841174 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.842156 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.842129 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-config\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.842156 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.842150 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b953752-e60a-4a77-bc61-fe0e1e78a022-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.842686 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.842662 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.842784 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.842686 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.842848 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.842808 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.843234 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.843212 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b953752-e60a-4a77-bc61-fe0e1e78a022-config-out\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.843604 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.843589 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-web-config\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:07.844919 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:07.844895 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47nt\" (UniqueName: \"kubernetes.io/projected/1b953752-e60a-4a77-bc61-fe0e1e78a022-kube-api-access-x47nt\") pod \"prometheus-k8s-0\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:08.020712 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:08.020676 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:08.202948 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:08.202915 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 00:13:08.206405 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:13:08.206361 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b953752_e60a_4a77_bc61_fe0e1e78a022.slice/crio-8331b3651b864e105c447e23508ac28be4eb1e1b659ad6a6b7debe239286e707 WatchSource:0}: Error finding container 8331b3651b864e105c447e23508ac28be4eb1e1b659ad6a6b7debe239286e707: Status 404 returned error can't find the container with id 8331b3651b864e105c447e23508ac28be4eb1e1b659ad6a6b7debe239286e707 Jul 23 00:13:08.496385 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:08.496352 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:13:08.554329 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:08.554290 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b953752-e60a-4a77-bc61-fe0e1e78a022","Type":"ContainerStarted","Data":"8331b3651b864e105c447e23508ac28be4eb1e1b659ad6a6b7debe239286e707"} Jul 23 00:13:09.497738 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:09.497708 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qwwgq" Jul 23 00:13:10.564338 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:10.564305 2555 generic.go:356] "Generic (PLEG): container finished" podID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerID="4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17" exitCode=0 Jul 23 00:13:10.564784 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:10.564404 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b953752-e60a-4a77-bc61-fe0e1e78a022","Type":"ContainerDied","Data":"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17"} Jul 23 00:13:11.978834 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:11.978805 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:12.079710 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:12.079670 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:12.462485 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:12.462459 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:13:13.031638 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:13.031563 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:13.088622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:13.088583 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:15.585983 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:15.585952 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b953752-e60a-4a77-bc61-fe0e1e78a022","Type":"ContainerStarted","Data":"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd"} Jul 23 00:13:15.586364 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:15.586013 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b953752-e60a-4a77-bc61-fe0e1e78a022","Type":"ContainerStarted","Data":"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe"} Jul 23 00:13:16.759012 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:16.758973 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:16.759513 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:16.759043 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:16.766004 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:16.765978 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:17.488584 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.488525 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5db59d744-b57zf" podUID="0a31e563-70c6-441d-bd67-cc3dadb8b3eb" containerName="registry" containerID="cri-o://f6348e97cf3a209e4ad507b41abaec18ef906635546d77b6fcf6a8ee289e3b7c" gracePeriod=30 Jul 23 00:13:17.598015 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.597994 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:13:17.648532 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.648509 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86c48b6cd6-r288z"] Jul 23 00:13:17.758916 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.758556 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:17.798874 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.798851 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:13:17.827579 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.827550 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:17.837316 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.837296 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-bound-sa-token\") pod \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " Jul 23 00:13:17.837434 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.837327 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-certificates\") pod \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " Jul 23 00:13:17.837434 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.837368 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khsk7\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-kube-api-access-khsk7\") pod \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " Jul 23 00:13:17.837434 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.837415 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls\") pod \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " Jul 23 00:13:17.837605 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.837441 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-image-registry-private-configuration\") pod \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " Jul 23 00:13:17.837605 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.837458 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-trusted-ca\") pod \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " Jul 23 00:13:17.837605 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.837498 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-installation-pull-secrets\") pod \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " Jul 23 00:13:17.837605 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.837521 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-ca-trust-extracted\") pod \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\" (UID: \"0a31e563-70c6-441d-bd67-cc3dadb8b3eb\") " Jul 23 00:13:17.837881 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.837847 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0a31e563-70c6-441d-bd67-cc3dadb8b3eb" (UID: "0a31e563-70c6-441d-bd67-cc3dadb8b3eb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:13:17.837962 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.837903 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0a31e563-70c6-441d-bd67-cc3dadb8b3eb" (UID: "0a31e563-70c6-441d-bd67-cc3dadb8b3eb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:13:17.840084 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.840052 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0a31e563-70c6-441d-bd67-cc3dadb8b3eb" (UID: "0a31e563-70c6-441d-bd67-cc3dadb8b3eb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:13:17.840173 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.840095 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0a31e563-70c6-441d-bd67-cc3dadb8b3eb" (UID: "0a31e563-70c6-441d-bd67-cc3dadb8b3eb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:13:17.840290 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.840263 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0a31e563-70c6-441d-bd67-cc3dadb8b3eb" (UID: "0a31e563-70c6-441d-bd67-cc3dadb8b3eb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:13:17.840341 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.840322 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0a31e563-70c6-441d-bd67-cc3dadb8b3eb" (UID: "0a31e563-70c6-441d-bd67-cc3dadb8b3eb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:13:17.840624 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.840607 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-kube-api-access-khsk7" (OuterVolumeSpecName: "kube-api-access-khsk7") pod "0a31e563-70c6-441d-bd67-cc3dadb8b3eb" (UID: "0a31e563-70c6-441d-bd67-cc3dadb8b3eb"). InnerVolumeSpecName "kube-api-access-khsk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:13:17.846419 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.846376 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0a31e563-70c6-441d-bd67-cc3dadb8b3eb" (UID: "0a31e563-70c6-441d-bd67-cc3dadb8b3eb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:13:17.938762 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.938739 2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:17.938762 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.938765 2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-image-registry-private-configuration\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:17.938917 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.938778 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-trusted-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:17.938917 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.938790 2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-installation-pull-secrets\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:17.938917 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.938803 2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-ca-trust-extracted\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:17.938917 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.938815 2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-bound-sa-token\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:17.938917 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.938826 2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-registry-certificates\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:17.938917 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.938838 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khsk7\" (UniqueName: \"kubernetes.io/projected/0a31e563-70c6-441d-bd67-cc3dadb8b3eb-kube-api-access-khsk7\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:17.972942 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:17.972918 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:18.074045 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:18.073960 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:18.603243 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:18.603211 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b953752-e60a-4a77-bc61-fe0e1e78a022","Type":"ContainerStarted","Data":"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2"} Jul 23 00:13:18.603458 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:18.603282 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b953752-e60a-4a77-bc61-fe0e1e78a022","Type":"ContainerStarted","Data":"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f"} Jul 23 00:13:18.603458 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:18.603323 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b953752-e60a-4a77-bc61-fe0e1e78a022","Type":"ContainerStarted","Data":"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862"} Jul 23 00:13:18.603458 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:18.603362 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b953752-e60a-4a77-bc61-fe0e1e78a022","Type":"ContainerStarted","Data":"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5"} Jul 23 00:13:18.604492 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:18.604470 2555 generic.go:356] "Generic (PLEG): container finished" podID="0a31e563-70c6-441d-bd67-cc3dadb8b3eb" containerID="f6348e97cf3a209e4ad507b41abaec18ef906635546d77b6fcf6a8ee289e3b7c" exitCode=0 Jul 23 00:13:18.604578 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:18.604502 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5db59d744-b57zf" Jul 23 00:13:18.604578 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:18.604550 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5db59d744-b57zf" event={"ID":"0a31e563-70c6-441d-bd67-cc3dadb8b3eb","Type":"ContainerDied","Data":"f6348e97cf3a209e4ad507b41abaec18ef906635546d77b6fcf6a8ee289e3b7c"} Jul 23 00:13:18.604644 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:18.604586 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5db59d744-b57zf" event={"ID":"0a31e563-70c6-441d-bd67-cc3dadb8b3eb","Type":"ContainerDied","Data":"93eeb941d698365ef4bac6613bb7140bbcd38cabfe6ca028ba93be3c3b93938a"} Jul 23 00:13:18.604644 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:18.604605 2555 scope.go:117] "RemoveContainer" containerID="f6348e97cf3a209e4ad507b41abaec18ef906635546d77b6fcf6a8ee289e3b7c" Jul 23 00:13:18.612560 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:18.612496 2555 scope.go:117] "RemoveContainer" containerID="f6348e97cf3a209e4ad507b41abaec18ef906635546d77b6fcf6a8ee289e3b7c" Jul 23 00:13:18.612765 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:13:18.612735 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6348e97cf3a209e4ad507b41abaec18ef906635546d77b6fcf6a8ee289e3b7c\": container with ID starting with f6348e97cf3a209e4ad507b41abaec18ef906635546d77b6fcf6a8ee289e3b7c not found: ID does not exist" containerID="f6348e97cf3a209e4ad507b41abaec18ef906635546d77b6fcf6a8ee289e3b7c" Jul 23 00:13:18.612811 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:18.612776 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6348e97cf3a209e4ad507b41abaec18ef906635546d77b6fcf6a8ee289e3b7c"} err="failed to get container status \"f6348e97cf3a209e4ad507b41abaec18ef906635546d77b6fcf6a8ee289e3b7c\": rpc error: code = NotFound desc = could not find container \"f6348e97cf3a209e4ad507b41abaec18ef906635546d77b6fcf6a8ee289e3b7c\": container with ID starting with f6348e97cf3a209e4ad507b41abaec18ef906635546d77b6fcf6a8ee289e3b7c not found: ID does not exist" Jul 23 00:13:18.631623 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:18.631557 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.392110037 podStartE2EDuration="11.631539893s" totalImagesPullingTime="8.216765763s" totalInitContainerRuntime="22.664093ms" isStatefulPod=false podCreationTimestamp="2026-07-23 00:13:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:13:18.629374383 +0000 UTC m=+318.590113014" watchObservedRunningTime="2026-07-23 00:13:18.631539893 +0000 UTC m=+318.592278514" Jul 23 00:13:18.641343 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:18.641324 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5db59d744-b57zf"] Jul 23 00:13:18.646165 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:18.646145 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5db59d744-b57zf"] Jul 23 00:13:20.506913 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:20.506886 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a31e563-70c6-441d-bd67-cc3dadb8b3eb" path="/var/lib/kubelet/pods/0a31e563-70c6-441d-bd67-cc3dadb8b3eb/volumes" Jul 23 00:13:21.029230 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:21.029195 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:21.089897 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:21.089866 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:22.009840 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:22.009803 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:22.058700 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:22.058669 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:23.021750 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:23.021712 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:13:25.634019 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:25.633936 2555 generic.go:356] "Generic (PLEG): container finished" podID="73b367b0-9408-4fcd-9535-cb198cfd7e34" containerID="93ded81b5fb7cab7b3c4c57173ede6579e6c6d70faf2b8f0b793ab6678131b98" exitCode=0 Jul 23 00:13:25.634379 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:25.634014 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm" event={"ID":"73b367b0-9408-4fcd-9535-cb198cfd7e34","Type":"ContainerDied","Data":"93ded81b5fb7cab7b3c4c57173ede6579e6c6d70faf2b8f0b793ab6678131b98"} Jul 23 00:13:25.634457 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:25.634376 2555 scope.go:117] "RemoveContainer" containerID="93ded81b5fb7cab7b3c4c57173ede6579e6c6d70faf2b8f0b793ab6678131b98" Jul 23 00:13:26.639195 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:26.639164 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-r7vwm" event={"ID":"73b367b0-9408-4fcd-9535-cb198cfd7e34","Type":"ContainerStarted","Data":"536d91d63292da14777b7085030abe1c765b50fa8fd8ff4f857ab353180a737d"} Jul 23 00:13:32.576943 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:32.576868 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f956d7d7b-64zg8" podUID="dd4f0e5f-b1d6-41a4-b135-22c806c131ee" containerName="console" containerID="cri-o://8c881c1517c4ff983b04a09054716516c0ee5985e169b3db94a42aa84e8762d4" gracePeriod=15 Jul 23 00:13:32.902438 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:32.902376 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:13:32.971467 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:32.969699 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-serving-cert\") pod \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " Jul 23 00:13:32.971467 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:32.969759 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-config\") pod \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " Jul 23 00:13:32.971467 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:32.969797 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-oauth-serving-cert\") pod \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " Jul 23 00:13:32.971467 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:32.969834 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-service-ca\") pod \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " Jul 23 00:13:32.971467 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:32.969894 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjqnj\" (UniqueName: \"kubernetes.io/projected/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-kube-api-access-rjqnj\") pod \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " Jul 23 00:13:32.971467 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:32.969932 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-trusted-ca-bundle\") pod \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " Jul 23 00:13:32.971467 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:32.969976 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-oauth-config\") pod \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\" (UID: \"dd4f0e5f-b1d6-41a4-b135-22c806c131ee\") " Jul 23 00:13:32.971467 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:32.971036 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dd4f0e5f-b1d6-41a4-b135-22c806c131ee" (UID: "dd4f0e5f-b1d6-41a4-b135-22c806c131ee"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:13:32.972482 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:32.972154 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dd4f0e5f-b1d6-41a4-b135-22c806c131ee" (UID: "dd4f0e5f-b1d6-41a4-b135-22c806c131ee"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:13:32.972649 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:32.972512 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-service-ca" (OuterVolumeSpecName: "service-ca") pod "dd4f0e5f-b1d6-41a4-b135-22c806c131ee" (UID: "dd4f0e5f-b1d6-41a4-b135-22c806c131ee"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:13:32.972720 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:32.972655 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-config" (OuterVolumeSpecName: "console-config") pod "dd4f0e5f-b1d6-41a4-b135-22c806c131ee" (UID: "dd4f0e5f-b1d6-41a4-b135-22c806c131ee"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:13:32.975689 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:32.975637 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dd4f0e5f-b1d6-41a4-b135-22c806c131ee" (UID: "dd4f0e5f-b1d6-41a4-b135-22c806c131ee"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:13:32.975805 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:32.975708 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-kube-api-access-rjqnj" (OuterVolumeSpecName: "kube-api-access-rjqnj") pod "dd4f0e5f-b1d6-41a4-b135-22c806c131ee" (UID: "dd4f0e5f-b1d6-41a4-b135-22c806c131ee"). InnerVolumeSpecName "kube-api-access-rjqnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:13:32.975921 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:32.975902 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dd4f0e5f-b1d6-41a4-b135-22c806c131ee" (UID: "dd4f0e5f-b1d6-41a4-b135-22c806c131ee"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:13:33.071822 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:33.071775 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-serving-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:33.071822 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:33.071822 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:33.072077 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:33.071838 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-oauth-serving-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:33.072077 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:33.071855 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:33.072077 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:33.071870 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjqnj\" (UniqueName: \"kubernetes.io/projected/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-kube-api-access-rjqnj\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:33.072077 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:33.071884 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-trusted-ca-bundle\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:33.072077 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:33.071899 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd4f0e5f-b1d6-41a4-b135-22c806c131ee-console-oauth-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:33.668819 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:33.668783 2555 generic.go:356] "Generic (PLEG): container finished" podID="dd4f0e5f-b1d6-41a4-b135-22c806c131ee" containerID="8c881c1517c4ff983b04a09054716516c0ee5985e169b3db94a42aa84e8762d4" exitCode=0 Jul 23 00:13:33.669253 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:33.668873 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f956d7d7b-64zg8" event={"ID":"dd4f0e5f-b1d6-41a4-b135-22c806c131ee","Type":"ContainerDied","Data":"8c881c1517c4ff983b04a09054716516c0ee5985e169b3db94a42aa84e8762d4"} Jul 23 00:13:33.669253 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:33.668881 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f956d7d7b-64zg8" Jul 23 00:13:33.669253 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:33.668924 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f956d7d7b-64zg8" event={"ID":"dd4f0e5f-b1d6-41a4-b135-22c806c131ee","Type":"ContainerDied","Data":"84404ab107a651056dfb6298b9e666d92f66d278b3eb54be6db0d5a0b6e1b1ec"} Jul 23 00:13:33.669253 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:33.668951 2555 scope.go:117] "RemoveContainer" containerID="8c881c1517c4ff983b04a09054716516c0ee5985e169b3db94a42aa84e8762d4" Jul 23 00:13:33.678757 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:33.678481 2555 scope.go:117] "RemoveContainer" containerID="8c881c1517c4ff983b04a09054716516c0ee5985e169b3db94a42aa84e8762d4" Jul 23 00:13:33.678899 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:13:33.678866 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c881c1517c4ff983b04a09054716516c0ee5985e169b3db94a42aa84e8762d4\": container with ID starting with 8c881c1517c4ff983b04a09054716516c0ee5985e169b3db94a42aa84e8762d4 not found: ID does not exist" containerID="8c881c1517c4ff983b04a09054716516c0ee5985e169b3db94a42aa84e8762d4" Jul 23 00:13:33.679154 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:33.678905 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c881c1517c4ff983b04a09054716516c0ee5985e169b3db94a42aa84e8762d4"} err="failed to get container status \"8c881c1517c4ff983b04a09054716516c0ee5985e169b3db94a42aa84e8762d4\": rpc error: code = NotFound desc = could not find container \"8c881c1517c4ff983b04a09054716516c0ee5985e169b3db94a42aa84e8762d4\": container with ID starting with 8c881c1517c4ff983b04a09054716516c0ee5985e169b3db94a42aa84e8762d4 not found: ID does not exist" Jul 23 00:13:33.693744 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:33.693706 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f956d7d7b-64zg8"] Jul 23 00:13:33.696427 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:33.696384 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f956d7d7b-64zg8"] Jul 23 00:13:34.502231 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:34.502193 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd4f0e5f-b1d6-41a4-b135-22c806c131ee" path="/var/lib/kubelet/pods/dd4f0e5f-b1d6-41a4-b135-22c806c131ee/volumes" Jul 23 00:13:34.880069 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:34.879969 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:34.976296 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:34.976266 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:13:42.676161 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:42.676097 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86c48b6cd6-r288z" podUID="950d50c5-3425-4b12-a39b-7d2adc7278fa" containerName="console" containerID="cri-o://b4403783f85f320cd943475a4332d5aedaa9b7e01132b1d3a68f69e195ac7646" gracePeriod=15 Jul 23 00:13:42.953501 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:42.953421 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:13:43.060517 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.060478 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-service-ca\") pod \"950d50c5-3425-4b12-a39b-7d2adc7278fa\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " Jul 23 00:13:43.060700 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.060580 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-oauth-serving-cert\") pod \"950d50c5-3425-4b12-a39b-7d2adc7278fa\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " Jul 23 00:13:43.060700 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.060623 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-serving-cert\") pod \"950d50c5-3425-4b12-a39b-7d2adc7278fa\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " Jul 23 00:13:43.060700 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.060647 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4z82\" (UniqueName: \"kubernetes.io/projected/950d50c5-3425-4b12-a39b-7d2adc7278fa-kube-api-access-v4z82\") pod \"950d50c5-3425-4b12-a39b-7d2adc7278fa\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " Jul 23 00:13:43.060700 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.060675 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-config\") pod \"950d50c5-3425-4b12-a39b-7d2adc7278fa\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " Jul 23 00:13:43.060906 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.060725 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-oauth-config\") pod \"950d50c5-3425-4b12-a39b-7d2adc7278fa\" (UID: \"950d50c5-3425-4b12-a39b-7d2adc7278fa\") " Jul 23 00:13:43.061034 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.061010 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-service-ca" (OuterVolumeSpecName: "service-ca") pod "950d50c5-3425-4b12-a39b-7d2adc7278fa" (UID: "950d50c5-3425-4b12-a39b-7d2adc7278fa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:13:43.061126 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.061051 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "950d50c5-3425-4b12-a39b-7d2adc7278fa" (UID: "950d50c5-3425-4b12-a39b-7d2adc7278fa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:13:43.061126 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.061075 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-config" (OuterVolumeSpecName: "console-config") pod "950d50c5-3425-4b12-a39b-7d2adc7278fa" (UID: "950d50c5-3425-4b12-a39b-7d2adc7278fa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:13:43.062856 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.062832 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950d50c5-3425-4b12-a39b-7d2adc7278fa-kube-api-access-v4z82" (OuterVolumeSpecName: "kube-api-access-v4z82") pod "950d50c5-3425-4b12-a39b-7d2adc7278fa" (UID: "950d50c5-3425-4b12-a39b-7d2adc7278fa"). InnerVolumeSpecName "kube-api-access-v4z82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:13:43.062937 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.062853 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "950d50c5-3425-4b12-a39b-7d2adc7278fa" (UID: "950d50c5-3425-4b12-a39b-7d2adc7278fa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:13:43.062973 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.062936 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "950d50c5-3425-4b12-a39b-7d2adc7278fa" (UID: "950d50c5-3425-4b12-a39b-7d2adc7278fa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:13:43.161559 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.161530 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:43.161559 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.161556 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-oauth-serving-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:43.161559 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.161566 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-serving-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:43.161769 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.161574 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4z82\" (UniqueName: \"kubernetes.io/projected/950d50c5-3425-4b12-a39b-7d2adc7278fa-kube-api-access-v4z82\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:43.161769 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.161583 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:43.161769 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.161593 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/950d50c5-3425-4b12-a39b-7d2adc7278fa-console-oauth-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:13:43.712835 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.712802 2555 generic.go:356] "Generic (PLEG): container finished" podID="950d50c5-3425-4b12-a39b-7d2adc7278fa" containerID="b4403783f85f320cd943475a4332d5aedaa9b7e01132b1d3a68f69e195ac7646" exitCode=0 Jul 23 00:13:43.713221 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.712846 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86c48b6cd6-r288z" Jul 23 00:13:43.713221 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.712889 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86c48b6cd6-r288z" event={"ID":"950d50c5-3425-4b12-a39b-7d2adc7278fa","Type":"ContainerDied","Data":"b4403783f85f320cd943475a4332d5aedaa9b7e01132b1d3a68f69e195ac7646"} Jul 23 00:13:43.713221 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.712936 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86c48b6cd6-r288z" event={"ID":"950d50c5-3425-4b12-a39b-7d2adc7278fa","Type":"ContainerDied","Data":"f2a60058c648d68abc79704ebe0b9829e2fbb3e65fe77aeb3fc41b6b89ec570e"} Jul 23 00:13:43.713221 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.712955 2555 scope.go:117] "RemoveContainer" containerID="b4403783f85f320cd943475a4332d5aedaa9b7e01132b1d3a68f69e195ac7646" Jul 23 00:13:43.721458 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.721442 2555 scope.go:117] "RemoveContainer" containerID="b4403783f85f320cd943475a4332d5aedaa9b7e01132b1d3a68f69e195ac7646" Jul 23 00:13:43.721709 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:13:43.721681 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4403783f85f320cd943475a4332d5aedaa9b7e01132b1d3a68f69e195ac7646\": container with ID starting with b4403783f85f320cd943475a4332d5aedaa9b7e01132b1d3a68f69e195ac7646 not found: ID does not exist" containerID="b4403783f85f320cd943475a4332d5aedaa9b7e01132b1d3a68f69e195ac7646" Jul 23 00:13:43.721801 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.721714 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4403783f85f320cd943475a4332d5aedaa9b7e01132b1d3a68f69e195ac7646"} err="failed to get container status \"b4403783f85f320cd943475a4332d5aedaa9b7e01132b1d3a68f69e195ac7646\": rpc error: code = NotFound desc = could not find container \"b4403783f85f320cd943475a4332d5aedaa9b7e01132b1d3a68f69e195ac7646\": container with ID starting with b4403783f85f320cd943475a4332d5aedaa9b7e01132b1d3a68f69e195ac7646 not found: ID does not exist" Jul 23 00:13:43.732015 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.731989 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86c48b6cd6-r288z"] Jul 23 00:13:43.735814 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:43.735790 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86c48b6cd6-r288z"] Jul 23 00:13:44.501405 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:13:44.501356 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="950d50c5-3425-4b12-a39b-7d2adc7278fa" path="/var/lib/kubelet/pods/950d50c5-3425-4b12-a39b-7d2adc7278fa/volumes" Jul 23 00:14:02.208176 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:02.208144 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:02.264853 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:02.264818 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:08.021098 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:08.021065 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:08.039079 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:08.039055 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:08.821879 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:08.821850 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:14.420897 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:14.420865 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs\") pod \"network-metrics-daemon-czq2t\" (UID: \"907325cd-64b7-4241-8969-a32a19bab628\") " pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:14:14.423152 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:14.423121 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907325cd-64b7-4241-8969-a32a19bab628-metrics-certs\") pod \"network-metrics-daemon-czq2t\" (UID: \"907325cd-64b7-4241-8969-a32a19bab628\") " pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:14:14.499220 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:14.499194 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pff25\"" Jul 23 00:14:14.507058 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:14.507037 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czq2t" Jul 23 00:14:14.631324 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:14.631298 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-czq2t"] Jul 23 00:14:14.633698 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:14:14.633673 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907325cd_64b7_4241_8969_a32a19bab628.slice/crio-22a8c0be407dc34399f15f17d34cec5067c59ffd42792d9b46b4786fd7b1875f WatchSource:0}: Error finding container 22a8c0be407dc34399f15f17d34cec5067c59ffd42792d9b46b4786fd7b1875f: Status 404 returned error can't find the container with id 22a8c0be407dc34399f15f17d34cec5067c59ffd42792d9b46b4786fd7b1875f Jul 23 00:14:14.830771 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:14.830688 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-czq2t" event={"ID":"907325cd-64b7-4241-8969-a32a19bab628","Type":"ContainerStarted","Data":"22a8c0be407dc34399f15f17d34cec5067c59ffd42792d9b46b4786fd7b1875f"} Jul 23 00:14:16.840181 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:16.840148 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-czq2t" event={"ID":"907325cd-64b7-4241-8969-a32a19bab628","Type":"ContainerStarted","Data":"38d0a6cf8adeee59badde8b3070e4b20df6cc7c3a50316b0c937881fb79c8d22"} Jul 23 00:14:16.840557 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:16.840199 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-czq2t" event={"ID":"907325cd-64b7-4241-8969-a32a19bab628","Type":"ContainerStarted","Data":"1a96dabf66efccf4fde56f7ee72c37a1e74b7328a841f80da97b9dae6797434f"} Jul 23 00:14:16.856123 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:16.856074 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-czq2t" podStartSLOduration=375.513271553 podStartE2EDuration="6m16.85605854s" totalImagesPullingTime="1.342786987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:14:16.855146389 +0000 UTC m=+376.815885007" watchObservedRunningTime="2026-07-23 00:14:16.85605854 +0000 UTC m=+376.816797158" Jul 23 00:14:16.945976 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:16.945948 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:17.021510 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:17.021472 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:17.076136 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:17.076108 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:20.710584 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:20.710542 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:20.915712 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:20.915680 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:22.358316 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:22.358262 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:22.417587 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:22.417563 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:25.671597 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:25.671568 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:26.116188 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.116103 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 00:14:26.116743 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.116689 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="prometheus" containerID="cri-o://a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe" gracePeriod=600 Jul 23 00:14:26.116743 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.116705 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="thanos-sidecar" containerID="cri-o://74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5" gracePeriod=600 Jul 23 00:14:26.117004 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.116760 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2" gracePeriod=600 Jul 23 00:14:26.117004 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.116803 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="kube-rbac-proxy-web" containerID="cri-o://599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862" gracePeriod=600 Jul 23 00:14:26.117004 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.116705 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="kube-rbac-proxy" containerID="cri-o://a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f" gracePeriod=600 Jul 23 00:14:26.117004 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.116867 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="config-reloader" containerID="cri-o://2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd" gracePeriod=600 Jul 23 00:14:26.392360 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.392325 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:26.527440 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527407 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.527440 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527446 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-2\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.527670 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527470 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x47nt\" (UniqueName: \"kubernetes.io/projected/1b953752-e60a-4a77-bc61-fe0e1e78a022-kube-api-access-x47nt\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.527670 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527489 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-config\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.527670 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527519 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-grpc-tls\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.527670 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527536 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-kube-rbac-proxy\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.527670 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527556 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-0\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.527670 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527588 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-1\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.527670 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527610 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-web-config\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.527670 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527656 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b953752-e60a-4a77-bc61-fe0e1e78a022-config-out\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.528146 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527682 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-tls\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.528146 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527720 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-metrics-client-ca\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.528146 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527742 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-trusted-ca-bundle\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.528146 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527780 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-serving-certs-ca-bundle\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.528146 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527805 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-metrics-client-certs\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.528146 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527841 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-db\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.528146 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527880 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-kubelet-serving-ca-bundle\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.528146 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527906 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b953752-e60a-4a77-bc61-fe0e1e78a022-tls-assets\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.528146 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527904 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:14:26.528146 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.527940 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-thanos-prometheus-http-client-file\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.528645 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.528226 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:14:26.528645 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.528273 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"1b953752-e60a-4a77-bc61-fe0e1e78a022\" (UID: \"1b953752-e60a-4a77-bc61-fe0e1e78a022\") " Jul 23 00:14:26.528645 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.528577 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.528645 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.528596 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.528914 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.528736 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:14:26.529858 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.529428 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:14:26.529858 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.529700 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:14:26.530037 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.529881 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:14:26.530248 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.530216 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:14:26.531218 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.530617 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:14:26.531218 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.530714 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:14:26.531218 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.530749 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b953752-e60a-4a77-bc61-fe0e1e78a022-kube-api-access-x47nt" (OuterVolumeSpecName: "kube-api-access-x47nt") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "kube-api-access-x47nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:14:26.531218 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.530786 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:14:26.531218 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.531148 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:14:26.531218 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.531182 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:14:26.532315 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.532283 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-config" (OuterVolumeSpecName: "config") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:14:26.532413 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.532309 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b953752-e60a-4a77-bc61-fe0e1e78a022-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:14:26.532466 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.532425 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:14:26.532617 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.532596 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:14:26.532663 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.532628 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:14:26.532734 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.532712 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b953752-e60a-4a77-bc61-fe0e1e78a022-config-out" (OuterVolumeSpecName: "config-out") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:14:26.542733 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.542714 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-web-config" (OuterVolumeSpecName: "web-config") pod "1b953752-e60a-4a77-bc61-fe0e1e78a022" (UID: "1b953752-e60a-4a77-bc61-fe0e1e78a022"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:14:26.629608 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629513 2555 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b953752-e60a-4a77-bc61-fe0e1e78a022-config-out\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.629608 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629554 2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.629608 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629566 2555 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-metrics-client-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.629608 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629576 2555 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.629608 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629586 2555 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-metrics-client-certs\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.629608 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629594 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-db\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.629608 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629603 2555 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.629608 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629612 2555 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b953752-e60a-4a77-bc61-fe0e1e78a022-tls-assets\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.629608 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629621 2555 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.630001 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629630 2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.630001 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629640 2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.630001 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629649 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x47nt\" (UniqueName: \"kubernetes.io/projected/1b953752-e60a-4a77-bc61-fe0e1e78a022-kube-api-access-x47nt\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.630001 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629657 2555 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.630001 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629665 2555 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-grpc-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.630001 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629672 2555 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-secret-kube-rbac-proxy\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.630001 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629680 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.630001 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629689 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1b953752-e60a-4a77-bc61-fe0e1e78a022-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.630001 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.629697 2555 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b953752-e60a-4a77-bc61-fe0e1e78a022-web-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:14:26.881709 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.881625 2555 generic.go:356] "Generic (PLEG): container finished" podID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerID="6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2" exitCode=0 Jul 23 00:14:26.881709 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.881650 2555 generic.go:356] "Generic (PLEG): container finished" podID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerID="a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f" exitCode=0 Jul 23 00:14:26.881709 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.881656 2555 generic.go:356] "Generic (PLEG): container finished" podID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerID="599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862" exitCode=0 Jul 23 00:14:26.881709 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.881662 2555 generic.go:356] "Generic (PLEG): container finished" podID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerID="74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5" exitCode=0 Jul 23 00:14:26.881709 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.881666 2555 generic.go:356] "Generic (PLEG): container finished" podID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerID="2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd" exitCode=0 Jul 23 00:14:26.881709 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.881671 2555 generic.go:356] "Generic (PLEG): container finished" podID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerID="a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe" exitCode=0 Jul 23 00:14:26.881709 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.881696 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:26.881709 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.881700 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b953752-e60a-4a77-bc61-fe0e1e78a022","Type":"ContainerDied","Data":"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2"} Jul 23 00:14:26.882460 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.881766 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b953752-e60a-4a77-bc61-fe0e1e78a022","Type":"ContainerDied","Data":"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f"} Jul 23 00:14:26.882460 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.881794 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b953752-e60a-4a77-bc61-fe0e1e78a022","Type":"ContainerDied","Data":"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862"} Jul 23 00:14:26.882460 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.881820 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b953752-e60a-4a77-bc61-fe0e1e78a022","Type":"ContainerDied","Data":"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5"} Jul 23 00:14:26.882460 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.881833 2555 scope.go:117] "RemoveContainer" containerID="6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2" Jul 23 00:14:26.882460 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.881844 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b953752-e60a-4a77-bc61-fe0e1e78a022","Type":"ContainerDied","Data":"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd"} Jul 23 00:14:26.882460 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.881868 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b953752-e60a-4a77-bc61-fe0e1e78a022","Type":"ContainerDied","Data":"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe"} Jul 23 00:14:26.882460 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.881892 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b953752-e60a-4a77-bc61-fe0e1e78a022","Type":"ContainerDied","Data":"8331b3651b864e105c447e23508ac28be4eb1e1b659ad6a6b7debe239286e707"} Jul 23 00:14:26.890644 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.890624 2555 scope.go:117] "RemoveContainer" containerID="a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f" Jul 23 00:14:26.897591 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.897571 2555 scope.go:117] "RemoveContainer" containerID="599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862" Jul 23 00:14:26.904414 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.904382 2555 scope.go:117] "RemoveContainer" containerID="74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5" Jul 23 00:14:26.907074 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.907052 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 00:14:26.912522 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.912093 2555 scope.go:117] "RemoveContainer" containerID="2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd" Jul 23 00:14:26.913702 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.913682 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 00:14:26.919364 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.919346 2555 scope.go:117] "RemoveContainer" containerID="a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe" Jul 23 00:14:26.926733 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.926707 2555 scope.go:117] "RemoveContainer" containerID="4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17" Jul 23 00:14:26.933457 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.933441 2555 scope.go:117] "RemoveContainer" containerID="6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2" Jul 23 00:14:26.933795 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:14:26.933771 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2\": container with ID starting with 6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2 not found: ID does not exist" containerID="6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2" Jul 23 00:14:26.933938 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.933803 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2"} err="failed to get container status \"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2\": rpc error: code = NotFound desc = could not find container \"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2\": container with ID starting with 6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2 not found: ID does not exist" Jul 23 00:14:26.933938 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.933832 2555 scope.go:117] "RemoveContainer" containerID="a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f" Jul 23 00:14:26.934443 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:14:26.934383 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f\": container with ID starting with a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f not found: ID does not exist" containerID="a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f" Jul 23 00:14:26.934561 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.934452 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f"} err="failed to get container status \"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f\": rpc error: code = NotFound desc = could not find container \"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f\": container with ID starting with a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f not found: ID does not exist" Jul 23 00:14:26.934561 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.934478 2555 scope.go:117] "RemoveContainer" containerID="599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862" Jul 23 00:14:26.934864 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:14:26.934845 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862\": container with ID starting with 599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862 not found: ID does not exist" containerID="599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862" Jul 23 00:14:26.934931 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.934870 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862"} err="failed to get container status \"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862\": rpc error: code = NotFound desc = could not find container \"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862\": container with ID starting with 599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862 not found: ID does not exist" Jul 23 00:14:26.934931 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.934886 2555 scope.go:117] "RemoveContainer" containerID="74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5" Jul 23 00:14:26.935142 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:14:26.935125 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5\": container with ID starting with 74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5 not found: ID does not exist" containerID="74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5" Jul 23 00:14:26.935198 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.935151 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5"} err="failed to get container status \"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5\": rpc error: code = NotFound desc = could not find container \"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5\": container with ID starting with 74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5 not found: ID does not exist" Jul 23 00:14:26.935198 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.935173 2555 scope.go:117] "RemoveContainer" containerID="2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd" Jul 23 00:14:26.935431 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:14:26.935378 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd\": container with ID starting with 2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd not found: ID does not exist" containerID="2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd" Jul 23 00:14:26.935508 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.935424 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd"} err="failed to get container status \"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd\": rpc error: code = NotFound desc = could not find container \"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd\": container with ID starting with 2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd not found: ID does not exist" Jul 23 00:14:26.935508 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.935448 2555 scope.go:117] "RemoveContainer" containerID="a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe" Jul 23 00:14:26.935707 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:14:26.935685 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe\": container with ID starting with a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe not found: ID does not exist" containerID="a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe" Jul 23 00:14:26.935744 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.935714 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe"} err="failed to get container status \"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe\": rpc error: code = NotFound desc = could not find container \"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe\": container with ID starting with a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe not found: ID does not exist" Jul 23 00:14:26.935744 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.935735 2555 scope.go:117] "RemoveContainer" containerID="4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17" Jul 23 00:14:26.935926 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.935911 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 00:14:26.935973 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:14:26.935954 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17\": container with ID starting with 4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17 not found: ID does not exist" containerID="4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17" Jul 23 00:14:26.936005 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.935973 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17"} err="failed to get container status \"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17\": rpc error: code = NotFound desc = could not find container \"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17\": container with ID starting with 4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17 not found: ID does not exist" Jul 23 00:14:26.936005 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.935992 2555 scope.go:117] "RemoveContainer" containerID="6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2" Jul 23 00:14:26.936240 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936220 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2"} err="failed to get container status \"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2\": rpc error: code = NotFound desc = could not find container \"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2\": container with ID starting with 6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2 not found: ID does not exist" Jul 23 00:14:26.936341 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936241 2555 scope.go:117] "RemoveContainer" containerID="a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f" Jul 23 00:14:26.936341 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936253 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="950d50c5-3425-4b12-a39b-7d2adc7278fa" containerName="console" Jul 23 00:14:26.936341 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936264 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="950d50c5-3425-4b12-a39b-7d2adc7278fa" containerName="console" Jul 23 00:14:26.936341 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936275 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="prometheus" Jul 23 00:14:26.936341 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936282 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="prometheus" Jul 23 00:14:26.936341 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936288 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="kube-rbac-proxy" Jul 23 00:14:26.936341 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936294 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="kube-rbac-proxy" Jul 23 00:14:26.936341 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936304 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd4f0e5f-b1d6-41a4-b135-22c806c131ee" containerName="console" Jul 23 00:14:26.936341 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936312 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd4f0e5f-b1d6-41a4-b135-22c806c131ee" containerName="console" Jul 23 00:14:26.936341 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936324 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="kube-rbac-proxy-web" Jul 23 00:14:26.936341 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936332 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="kube-rbac-proxy-web" Jul 23 00:14:26.936341 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936342 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a31e563-70c6-441d-bd67-cc3dadb8b3eb" containerName="registry" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936351 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a31e563-70c6-441d-bd67-cc3dadb8b3eb" containerName="registry" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936364 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="thanos-sidecar" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936370 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="thanos-sidecar" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936381 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="config-reloader" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936404 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="config-reloader" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936415 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="init-config-reloader" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936424 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="init-config-reloader" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936435 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="kube-rbac-proxy-thanos" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936441 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="kube-rbac-proxy-thanos" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936502 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a31e563-70c6-441d-bd67-cc3dadb8b3eb" containerName="registry" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936511 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="thanos-sidecar" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936516 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="950d50c5-3425-4b12-a39b-7d2adc7278fa" containerName="console" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936522 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="config-reloader" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936519 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f"} err="failed to get container status \"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f\": rpc error: code = NotFound desc = could not find container \"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f\": container with ID starting with a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f not found: ID does not exist" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936547 2555 scope.go:117] "RemoveContainer" containerID="599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936530 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="kube-rbac-proxy-thanos" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936615 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="kube-rbac-proxy-web" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936627 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="prometheus" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936637 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd4f0e5f-b1d6-41a4-b135-22c806c131ee" containerName="console" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936647 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" containerName="kube-rbac-proxy" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936822 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862"} err="failed to get container status \"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862\": rpc error: code = NotFound desc = could not find container \"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862\": container with ID starting with 599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862 not found: ID does not exist" Jul 23 00:14:26.936893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.936841 2555 scope.go:117] "RemoveContainer" containerID="74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5" Jul 23 00:14:26.937883 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.937055 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5"} err="failed to get container status \"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5\": rpc error: code = NotFound desc = could not find container \"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5\": container with ID starting with 74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5 not found: ID does not exist" Jul 23 00:14:26.937883 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.937072 2555 scope.go:117] "RemoveContainer" containerID="2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd" Jul 23 00:14:26.937883 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.937268 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd"} err="failed to get container status \"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd\": rpc error: code = NotFound desc = could not find container \"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd\": container with ID starting with 2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd not found: ID does not exist" Jul 23 00:14:26.937883 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.937283 2555 scope.go:117] "RemoveContainer" containerID="a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe" Jul 23 00:14:26.937883 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.937540 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe"} err="failed to get container status \"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe\": rpc error: code = NotFound desc = could not find container \"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe\": container with ID starting with a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe not found: ID does not exist" Jul 23 00:14:26.937883 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.937555 2555 scope.go:117] "RemoveContainer" containerID="4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17" Jul 23 00:14:26.937883 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.937751 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17"} err="failed to get container status \"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17\": rpc error: code = NotFound desc = could not find container \"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17\": container with ID starting with 4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17 not found: ID does not exist" Jul 23 00:14:26.937883 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.937766 2555 scope.go:117] "RemoveContainer" containerID="6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2" Jul 23 00:14:26.938253 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.937957 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2"} err="failed to get container status \"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2\": rpc error: code = NotFound desc = could not find container \"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2\": container with ID starting with 6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2 not found: ID does not exist" Jul 23 00:14:26.938253 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.937971 2555 scope.go:117] "RemoveContainer" containerID="a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f" Jul 23 00:14:26.938253 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.938229 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f"} err="failed to get container status \"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f\": rpc error: code = NotFound desc = could not find container \"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f\": container with ID starting with a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f not found: ID does not exist" Jul 23 00:14:26.938253 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.938247 2555 scope.go:117] "RemoveContainer" containerID="599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862" Jul 23 00:14:26.938615 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.938594 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862"} err="failed to get container status \"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862\": rpc error: code = NotFound desc = could not find container \"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862\": container with ID starting with 599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862 not found: ID does not exist" Jul 23 00:14:26.938615 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.938613 2555 scope.go:117] "RemoveContainer" containerID="74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5" Jul 23 00:14:26.938898 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.938876 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5"} err="failed to get container status \"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5\": rpc error: code = NotFound desc = could not find container \"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5\": container with ID starting with 74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5 not found: ID does not exist" Jul 23 00:14:26.938965 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.938900 2555 scope.go:117] "RemoveContainer" containerID="2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd" Jul 23 00:14:26.939172 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.939151 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd"} err="failed to get container status \"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd\": rpc error: code = NotFound desc = could not find container \"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd\": container with ID starting with 2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd not found: ID does not exist" Jul 23 00:14:26.939212 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.939175 2555 scope.go:117] "RemoveContainer" containerID="a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe" Jul 23 00:14:26.939421 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.939380 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe"} err="failed to get container status \"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe\": rpc error: code = NotFound desc = could not find container \"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe\": container with ID starting with a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe not found: ID does not exist" Jul 23 00:14:26.939511 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.939421 2555 scope.go:117] "RemoveContainer" containerID="4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17" Jul 23 00:14:26.939677 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.939641 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17"} err="failed to get container status \"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17\": rpc error: code = NotFound desc = could not find container \"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17\": container with ID starting with 4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17 not found: ID does not exist" Jul 23 00:14:26.939677 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.939664 2555 scope.go:117] "RemoveContainer" containerID="6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2" Jul 23 00:14:26.939861 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.939845 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2"} err="failed to get container status \"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2\": rpc error: code = NotFound desc = could not find container \"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2\": container with ID starting with 6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2 not found: ID does not exist" Jul 23 00:14:26.939901 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.939862 2555 scope.go:117] "RemoveContainer" containerID="a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f" Jul 23 00:14:26.940055 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.940036 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f"} err="failed to get container status \"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f\": rpc error: code = NotFound desc = could not find container \"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f\": container with ID starting with a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f not found: ID does not exist" Jul 23 00:14:26.940148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.940058 2555 scope.go:117] "RemoveContainer" containerID="599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862" Jul 23 00:14:26.940289 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.940274 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862"} err="failed to get container status \"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862\": rpc error: code = NotFound desc = could not find container \"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862\": container with ID starting with 599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862 not found: ID does not exist" Jul 23 00:14:26.940338 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.940289 2555 scope.go:117] "RemoveContainer" containerID="74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5" Jul 23 00:14:26.940492 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.940474 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5"} err="failed to get container status \"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5\": rpc error: code = NotFound desc = could not find container \"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5\": container with ID starting with 74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5 not found: ID does not exist" Jul 23 00:14:26.940572 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.940492 2555 scope.go:117] "RemoveContainer" containerID="2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd" Jul 23 00:14:26.940725 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.940708 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd"} err="failed to get container status \"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd\": rpc error: code = NotFound desc = could not find container \"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd\": container with ID starting with 2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd not found: ID does not exist" Jul 23 00:14:26.940777 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.940725 2555 scope.go:117] "RemoveContainer" containerID="a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe" Jul 23 00:14:26.940910 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.940893 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe"} err="failed to get container status \"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe\": rpc error: code = NotFound desc = could not find container \"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe\": container with ID starting with a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe not found: ID does not exist" Jul 23 00:14:26.940970 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.940911 2555 scope.go:117] "RemoveContainer" containerID="4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17" Jul 23 00:14:26.941105 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.941092 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17"} err="failed to get container status \"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17\": rpc error: code = NotFound desc = could not find container \"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17\": container with ID starting with 4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17 not found: ID does not exist" Jul 23 00:14:26.941146 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.941105 2555 scope.go:117] "RemoveContainer" containerID="6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2" Jul 23 00:14:26.941309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.941287 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2"} err="failed to get container status \"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2\": rpc error: code = NotFound desc = could not find container \"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2\": container with ID starting with 6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2 not found: ID does not exist" Jul 23 00:14:26.941349 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.941309 2555 scope.go:117] "RemoveContainer" containerID="a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f" Jul 23 00:14:26.941515 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.941498 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f"} err="failed to get container status \"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f\": rpc error: code = NotFound desc = could not find container \"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f\": container with ID starting with a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f not found: ID does not exist" Jul 23 00:14:26.941585 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.941516 2555 scope.go:117] "RemoveContainer" containerID="599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862" Jul 23 00:14:26.941991 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.941969 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862"} err="failed to get container status \"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862\": rpc error: code = NotFound desc = could not find container \"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862\": container with ID starting with 599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862 not found: ID does not exist" Jul 23 00:14:26.942068 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.941992 2555 scope.go:117] "RemoveContainer" containerID="74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5" Jul 23 00:14:26.942250 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.942223 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5"} err="failed to get container status \"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5\": rpc error: code = NotFound desc = could not find container \"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5\": container with ID starting with 74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5 not found: ID does not exist" Jul 23 00:14:26.942331 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.942251 2555 scope.go:117] "RemoveContainer" containerID="2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd" Jul 23 00:14:26.942443 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.942423 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:26.942555 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.942525 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd"} err="failed to get container status \"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd\": rpc error: code = NotFound desc = could not find container \"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd\": container with ID starting with 2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd not found: ID does not exist" Jul 23 00:14:26.942555 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.942546 2555 scope.go:117] "RemoveContainer" containerID="a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe" Jul 23 00:14:26.943115 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.943090 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe"} err="failed to get container status \"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe\": rpc error: code = NotFound desc = could not find container \"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe\": container with ID starting with a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe not found: ID does not exist" Jul 23 00:14:26.943215 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.943114 2555 scope.go:117] "RemoveContainer" containerID="4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17" Jul 23 00:14:26.943443 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.943418 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17"} err="failed to get container status \"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17\": rpc error: code = NotFound desc = could not find container \"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17\": container with ID starting with 4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17 not found: ID does not exist" Jul 23 00:14:26.943517 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.943447 2555 scope.go:117] "RemoveContainer" containerID="6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2" Jul 23 00:14:26.943695 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.943677 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2"} err="failed to get container status \"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2\": rpc error: code = NotFound desc = could not find container \"6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2\": container with ID starting with 6c546e5507278854c8c160363f6af79a4206d73ae673ca533302503e6abb6bf2 not found: ID does not exist" Jul 23 00:14:26.943804 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.943698 2555 scope.go:117] "RemoveContainer" containerID="a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f" Jul 23 00:14:26.943933 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.943916 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f"} err="failed to get container status \"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f\": rpc error: code = NotFound desc = could not find container \"a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f\": container with ID starting with a0663697def1a285a738af69622e43d5050700811b2fd6984ec5df29648ef87f not found: ID does not exist" Jul 23 00:14:26.943975 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.943934 2555 scope.go:117] "RemoveContainer" containerID="599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862" Jul 23 00:14:26.944116 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.944098 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862"} err="failed to get container status \"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862\": rpc error: code = NotFound desc = could not find container \"599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862\": container with ID starting with 599c75c127292b3eebbe13e1598336cf54eb745e032a1e67d9f99b9bc5778862 not found: ID does not exist" Jul 23 00:14:26.944190 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.944117 2555 scope.go:117] "RemoveContainer" containerID="74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5" Jul 23 00:14:26.944305 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.944290 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5"} err="failed to get container status \"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5\": rpc error: code = NotFound desc = could not find container \"74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5\": container with ID starting with 74081ab745f035e9296905363e9f2c5e97f79a737f14a36ddf64351daa9e4cb5 not found: ID does not exist" Jul 23 00:14:26.944356 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.944305 2555 scope.go:117] "RemoveContainer" containerID="2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd" Jul 23 00:14:26.944568 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.944549 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd"} err="failed to get container status \"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd\": rpc error: code = NotFound desc = could not find container \"2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd\": container with ID starting with 2de645ea234415e98451460c4857b92bdbaf4b2bfbdc78ae42317402c38903cd not found: ID does not exist" Jul 23 00:14:26.944646 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.944569 2555 scope.go:117] "RemoveContainer" containerID="a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe" Jul 23 00:14:26.944797 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.944778 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe"} err="failed to get container status \"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe\": rpc error: code = NotFound desc = could not find container \"a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe\": container with ID starting with a4b9d3e5b368c33c2c03da43c99a179f20aefd46d3d2e10bc5e09b903c7bccfe not found: ID does not exist" Jul 23 00:14:26.944858 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.944797 2555 scope.go:117] "RemoveContainer" containerID="4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17" Jul 23 00:14:26.945034 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.945007 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17"} err="failed to get container status \"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17\": rpc error: code = NotFound desc = could not find container \"4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17\": container with ID starting with 4fa53844e5209edaeec7e12f69d21c7c4a7dfa860e9d25eacfb315b45a5d4e17 not found: ID does not exist" Jul 23 00:14:26.945148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.945058 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 23 00:14:26.945148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.945024 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 23 00:14:26.945148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.945120 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 23 00:14:26.945286 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.945173 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 23 00:14:26.945470 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.945453 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 23 00:14:26.945531 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.945486 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-h6p4b\"" Jul 23 00:14:26.945608 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.945588 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 23 00:14:26.945659 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.945614 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 23 00:14:26.945659 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.945617 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 23 00:14:26.945659 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.945637 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 23 00:14:26.945801 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.945716 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 23 00:14:26.945801 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.945769 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 23 00:14:26.945891 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.945787 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 23 00:14:26.946015 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.946001 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9usec5k7h73hk\"" Jul 23 00:14:26.946059 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.946027 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 23 00:14:26.947895 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.947878 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 23 00:14:26.950232 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.950214 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 23 00:14:26.954436 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:26.954413 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 00:14:27.033097 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033063 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033097 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033106 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033291 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033165 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033291 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033216 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033291 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033266 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033385 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033293 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-web-config\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033385 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033313 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033457 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033380 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033457 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033428 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zhr7\" (UniqueName: \"kubernetes.io/projected/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-kube-api-access-8zhr7\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033457 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033449 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033556 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033467 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033556 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033498 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033556 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033516 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033556 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033530 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033702 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033557 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033702 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033578 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-config\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033702 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033595 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-config-out\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033702 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033615 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033702 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033652 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.033866 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.033702 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.134151 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.134116 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.134266 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.134170 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-config\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.134266 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.134199 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-config-out\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.134426 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.134401 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.134539 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.134495 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.136148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.134661 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.136148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.134749 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.136148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.134777 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.136148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.134832 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.136148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.134856 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.136148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.134898 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.136148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.134928 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-web-config\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.136148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.135279 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.136148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.135326 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.136148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.135340 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.136148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.135366 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zhr7\" (UniqueName: \"kubernetes.io/projected/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-kube-api-access-8zhr7\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.137047 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.136245 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.137047 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.136760 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.137047 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.136804 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.137047 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.136834 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.137047 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.136879 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.137047 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.136921 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.137047 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.136961 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.138598 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.138287 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.138598 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.138402 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.139222 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.138939 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.139222 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.138981 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-web-config\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.139474 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.139318 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-config-out\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.141128 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.140111 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.141128 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.140548 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.141128 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.140566 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.141128 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.141010 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.141128 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.141043 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.141434 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.141162 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.141434 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.141231 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.141434 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.141321 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.141575 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.141479 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-config\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.141829 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.141808 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.142443 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.142422 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.143491 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.143469 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zhr7\" (UniqueName: \"kubernetes.io/projected/eeededa1-ed50-413b-b0cd-08e0eb02d7ee-kube-api-access-8zhr7\") pod \"prometheus-k8s-0\" (UID: \"eeededa1-ed50-413b-b0cd-08e0eb02d7ee\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.252689 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.252650 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:27.393907 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.393884 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 23 00:14:27.396198 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:14:27.396176 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeededa1_ed50_413b_b0cd_08e0eb02d7ee.slice/crio-d48db2bbb98aeb3aae14cbfbe2a360475971d30fa43949a8d80be78faf5c1098 WatchSource:0}: Error finding container d48db2bbb98aeb3aae14cbfbe2a360475971d30fa43949a8d80be78faf5c1098: Status 404 returned error can't find the container with id d48db2bbb98aeb3aae14cbfbe2a360475971d30fa43949a8d80be78faf5c1098 Jul 23 00:14:27.886852 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.886818 2555 generic.go:356] "Generic (PLEG): container finished" podID="eeededa1-ed50-413b-b0cd-08e0eb02d7ee" containerID="0879392c49c2ac433bd5045cce4ee3a6b2bead85dd68adec6dfeb75210ac9d98" exitCode=0 Jul 23 00:14:27.887321 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.886902 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeededa1-ed50-413b-b0cd-08e0eb02d7ee","Type":"ContainerDied","Data":"0879392c49c2ac433bd5045cce4ee3a6b2bead85dd68adec6dfeb75210ac9d98"} Jul 23 00:14:27.887321 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:27.886973 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeededa1-ed50-413b-b0cd-08e0eb02d7ee","Type":"ContainerStarted","Data":"d48db2bbb98aeb3aae14cbfbe2a360475971d30fa43949a8d80be78faf5c1098"} Jul 23 00:14:28.502578 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:28.502547 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b953752-e60a-4a77-bc61-fe0e1e78a022" path="/var/lib/kubelet/pods/1b953752-e60a-4a77-bc61-fe0e1e78a022/volumes" Jul 23 00:14:28.895733 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:28.895649 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeededa1-ed50-413b-b0cd-08e0eb02d7ee","Type":"ContainerStarted","Data":"01d096e2da88f813cd0019741dde5618635bf572127018b79717088680091400"} Jul 23 00:14:28.895733 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:28.895711 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeededa1-ed50-413b-b0cd-08e0eb02d7ee","Type":"ContainerStarted","Data":"9e73fa16ced955ce682f892e614fe2cec4fa4641d86ff3bff27790d1f261153b"} Jul 23 00:14:28.896214 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:28.895755 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeededa1-ed50-413b-b0cd-08e0eb02d7ee","Type":"ContainerStarted","Data":"859ae9f12eafb96ce35d5dbe0c63d6fef52aac2ca552ddedd0931bd470329f64"} Jul 23 00:14:28.896214 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:28.895780 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeededa1-ed50-413b-b0cd-08e0eb02d7ee","Type":"ContainerStarted","Data":"a262708c0ab65415967f0ca43a54cadd810dc2dc17e82c611df1af7960e38493"} Jul 23 00:14:28.896214 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:28.895803 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeededa1-ed50-413b-b0cd-08e0eb02d7ee","Type":"ContainerStarted","Data":"98eca7ba4952401ec2a2e47459e090d7cb9fd3d4c08759fe98258c476e8c3d5c"} Jul 23 00:14:28.896214 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:28.895824 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eeededa1-ed50-413b-b0cd-08e0eb02d7ee","Type":"ContainerStarted","Data":"07e3e842e27b1e6ebc8d875df2a4b14a62aab5156f4d1fae19492aa9804e8f65"} Jul 23 00:14:28.926022 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:28.925967 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.904517024 podStartE2EDuration="2.925949756s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.432732ms" isStatefulPod=false podCreationTimestamp="2026-07-23 00:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:14:28.923985365 +0000 UTC m=+388.884723976" watchObservedRunningTime="2026-07-23 00:14:28.925949756 +0000 UTC m=+388.886688401" Jul 23 00:14:30.077567 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.077520 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-db9cdddf4-t9hmp"] Jul 23 00:14:30.082881 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.082853 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.091931 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.091581 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-db9cdddf4-t9hmp"] Jul 23 00:14:30.128611 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.128572 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:30.228759 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.228723 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:30.265959 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.265921 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-service-ca\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.265959 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.265964 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh8mc\" (UniqueName: \"kubernetes.io/projected/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-kube-api-access-kh8mc\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.266164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.265985 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-trusted-ca-bundle\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.266164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.266113 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-oauth-config\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.266261 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.266176 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-serving-cert\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.266261 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.266219 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-config\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.266261 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.266252 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-oauth-serving-cert\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.328734 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.328641 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:30.366826 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.366789 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-oauth-serving-cert\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.367004 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.366848 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-service-ca\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.367004 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.366879 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh8mc\" (UniqueName: \"kubernetes.io/projected/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-kube-api-access-kh8mc\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.367004 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.366896 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-trusted-ca-bundle\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.367004 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.366929 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-oauth-config\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.367004 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.366967 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-serving-cert\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.367004 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.367003 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-config\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.367654 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.367631 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-oauth-serving-cert\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.367760 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.367701 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-service-ca\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.367806 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.367790 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-config\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.367873 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.367854 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-trusted-ca-bundle\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.369475 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.369453 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-oauth-config\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.369559 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.369478 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-serving-cert\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.374637 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.374618 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh8mc\" (UniqueName: \"kubernetes.io/projected/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-kube-api-access-kh8mc\") pod \"console-db9cdddf4-t9hmp\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.394295 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.394267 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:30.527930 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.527900 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-db9cdddf4-t9hmp"] Jul 23 00:14:30.531143 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:14:30.531113 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b2e658_ec91_4e9e_93db_0f6f3eab90d1.slice/crio-1544632786fd28cc88c9b98fb99fd40a2836911aab3513d141363b31ac1a843b WatchSource:0}: Error finding container 1544632786fd28cc88c9b98fb99fd40a2836911aab3513d141363b31ac1a843b: Status 404 returned error can't find the container with id 1544632786fd28cc88c9b98fb99fd40a2836911aab3513d141363b31ac1a843b Jul 23 00:14:30.881622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.881539 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:30.905513 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.905482 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db9cdddf4-t9hmp" event={"ID":"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1","Type":"ContainerStarted","Data":"f7b75ca0c59cebf0ba8418c211d3ae4df56e518c70fd5a1bc54e98cc4efa7fd3"} Jul 23 00:14:30.905669 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.905525 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db9cdddf4-t9hmp" event={"ID":"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1","Type":"ContainerStarted","Data":"1544632786fd28cc88c9b98fb99fd40a2836911aab3513d141363b31ac1a843b"} Jul 23 00:14:30.926843 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.926793 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-db9cdddf4-t9hmp" podStartSLOduration=0.926776432 podStartE2EDuration="926.776432ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:14:30.925455841 +0000 UTC m=+390.886194459" watchObservedRunningTime="2026-07-23 00:14:30.926776432 +0000 UTC m=+390.887515050" Jul 23 00:14:30.982334 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:30.982305 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:32.252798 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:32.252749 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:14:40.394925 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:40.394882 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:40.395444 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:40.394961 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:40.399875 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:40.399849 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:40.948547 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:40.948516 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:14:40.990806 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:40.990775 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b59545fd4-ckp28"] Jul 23 00:14:41.102258 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:41.102224 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:41.154457 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:41.154418 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:41.207653 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:41.207567 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:14:51.262576 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:14:51.262539 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:15:06.017230 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.017165 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b59545fd4-ckp28" podUID="d5c1e6cf-5ef3-4f78-94a6-f564ae3af861" containerName="console" containerID="cri-o://3c7191808329331ffb9d9ff86ead62c3bb7195f819766f0a18b0425d975b55b4" gracePeriod=15 Jul 23 00:15:06.292275 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.292211 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:15:06.381109 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.381069 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bcxl\" (UniqueName: \"kubernetes.io/projected/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-kube-api-access-4bcxl\") pod \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " Jul 23 00:15:06.381109 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.381110 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-oauth-config\") pod \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " Jul 23 00:15:06.381350 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.381176 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-service-ca\") pod \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " Jul 23 00:15:06.381350 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.381214 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-config\") pod \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " Jul 23 00:15:06.381350 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.381239 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-trusted-ca-bundle\") pod \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " Jul 23 00:15:06.381350 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.381263 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-oauth-serving-cert\") pod \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " Jul 23 00:15:06.381350 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.381296 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-serving-cert\") pod \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\" (UID: \"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861\") " Jul 23 00:15:06.381641 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.381610 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-service-ca" (OuterVolumeSpecName: "service-ca") pod "d5c1e6cf-5ef3-4f78-94a6-f564ae3af861" (UID: "d5c1e6cf-5ef3-4f78-94a6-f564ae3af861"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:15:06.381711 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.381680 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d5c1e6cf-5ef3-4f78-94a6-f564ae3af861" (UID: "d5c1e6cf-5ef3-4f78-94a6-f564ae3af861"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:15:06.381711 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.381694 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-config" (OuterVolumeSpecName: "console-config") pod "d5c1e6cf-5ef3-4f78-94a6-f564ae3af861" (UID: "d5c1e6cf-5ef3-4f78-94a6-f564ae3af861"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:15:06.381809 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.381773 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d5c1e6cf-5ef3-4f78-94a6-f564ae3af861" (UID: "d5c1e6cf-5ef3-4f78-94a6-f564ae3af861"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:15:06.383382 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.383354 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d5c1e6cf-5ef3-4f78-94a6-f564ae3af861" (UID: "d5c1e6cf-5ef3-4f78-94a6-f564ae3af861"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:15:06.383382 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.383371 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-kube-api-access-4bcxl" (OuterVolumeSpecName: "kube-api-access-4bcxl") pod "d5c1e6cf-5ef3-4f78-94a6-f564ae3af861" (UID: "d5c1e6cf-5ef3-4f78-94a6-f564ae3af861"). InnerVolumeSpecName "kube-api-access-4bcxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:15:06.383382 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.383409 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d5c1e6cf-5ef3-4f78-94a6-f564ae3af861" (UID: "d5c1e6cf-5ef3-4f78-94a6-f564ae3af861"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:15:06.482235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.482188 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:15:06.482235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.482228 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:15:06.482235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.482239 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-trusted-ca-bundle\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:15:06.482235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.482248 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-oauth-serving-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:15:06.482235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.482256 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-serving-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:15:06.482585 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.482265 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bcxl\" (UniqueName: \"kubernetes.io/projected/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-kube-api-access-4bcxl\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:15:06.482585 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:06.482274 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861-console-oauth-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:15:07.059685 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:07.059650 2555 generic.go:356] "Generic (PLEG): container finished" podID="d5c1e6cf-5ef3-4f78-94a6-f564ae3af861" containerID="3c7191808329331ffb9d9ff86ead62c3bb7195f819766f0a18b0425d975b55b4" exitCode=0 Jul 23 00:15:07.060085 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:07.059696 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b59545fd4-ckp28" Jul 23 00:15:07.060085 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:07.059732 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b59545fd4-ckp28" event={"ID":"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861","Type":"ContainerDied","Data":"3c7191808329331ffb9d9ff86ead62c3bb7195f819766f0a18b0425d975b55b4"} Jul 23 00:15:07.060085 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:07.059787 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b59545fd4-ckp28" event={"ID":"d5c1e6cf-5ef3-4f78-94a6-f564ae3af861","Type":"ContainerDied","Data":"f03073df5f3eaf423497965a5a1ce5a6279dea07ac12e7e73af50bca2495dee5"} Jul 23 00:15:07.060085 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:07.059807 2555 scope.go:117] "RemoveContainer" containerID="3c7191808329331ffb9d9ff86ead62c3bb7195f819766f0a18b0425d975b55b4" Jul 23 00:15:07.068246 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:07.068228 2555 scope.go:117] "RemoveContainer" containerID="3c7191808329331ffb9d9ff86ead62c3bb7195f819766f0a18b0425d975b55b4" Jul 23 00:15:07.068663 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:15:07.068646 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7191808329331ffb9d9ff86ead62c3bb7195f819766f0a18b0425d975b55b4\": container with ID starting with 3c7191808329331ffb9d9ff86ead62c3bb7195f819766f0a18b0425d975b55b4 not found: ID does not exist" containerID="3c7191808329331ffb9d9ff86ead62c3bb7195f819766f0a18b0425d975b55b4" Jul 23 00:15:07.068713 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:07.068674 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7191808329331ffb9d9ff86ead62c3bb7195f819766f0a18b0425d975b55b4"} err="failed to get container status \"3c7191808329331ffb9d9ff86ead62c3bb7195f819766f0a18b0425d975b55b4\": rpc error: code = NotFound desc = could not find container \"3c7191808329331ffb9d9ff86ead62c3bb7195f819766f0a18b0425d975b55b4\": container with ID starting with 3c7191808329331ffb9d9ff86ead62c3bb7195f819766f0a18b0425d975b55b4 not found: ID does not exist" Jul 23 00:15:07.075338 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:07.075314 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b59545fd4-ckp28"] Jul 23 00:15:07.076597 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:07.076578 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b59545fd4-ckp28"] Jul 23 00:15:08.501641 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:08.501607 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5c1e6cf-5ef3-4f78-94a6-f564ae3af861" path="/var/lib/kubelet/pods/d5c1e6cf-5ef3-4f78-94a6-f564ae3af861/volumes" Jul 23 00:15:27.253116 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:27.253029 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:15:27.268911 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:27.268886 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:15:28.157106 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:28.157074 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 23 00:15:35.956864 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:35.956830 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:15:36.012756 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:36.012722 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:15:47.902034 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:47.901999 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:15:48.003887 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:48.003853 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:15:50.999554 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:50.999518 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:15:51.099852 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:51.099820 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:15:51.267152 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:51.267056 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:15:55.216981 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.216942 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fbb89dcd9-hfd9r"] Jul 23 00:15:55.217408 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.217271 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c1e6cf-5ef3-4f78-94a6-f564ae3af861" containerName="console" Jul 23 00:15:55.217408 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.217286 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c1e6cf-5ef3-4f78-94a6-f564ae3af861" containerName="console" Jul 23 00:15:55.217408 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.217355 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c1e6cf-5ef3-4f78-94a6-f564ae3af861" containerName="console" Jul 23 00:15:55.220231 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.220213 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.230216 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.230192 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fbb89dcd9-hfd9r"] Jul 23 00:15:55.266299 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.266272 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:15:55.300821 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.300793 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-config\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.300976 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.300830 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7p6q\" (UniqueName: \"kubernetes.io/projected/39bbee47-517e-4777-90b9-a6bb9be74ddf-kube-api-access-r7p6q\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.300976 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.300852 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-oauth-serving-cert\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.300976 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.300875 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-oauth-config\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.300976 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.300938 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-serving-cert\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.300976 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.300961 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-trusted-ca-bundle\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.301172 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.300982 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-service-ca\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.368036 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.367997 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:15:55.402228 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.402199 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-config\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.402355 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.402233 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7p6q\" (UniqueName: \"kubernetes.io/projected/39bbee47-517e-4777-90b9-a6bb9be74ddf-kube-api-access-r7p6q\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.402355 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.402261 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-oauth-serving-cert\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.402355 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.402296 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-oauth-config\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.402355 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.402339 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-serving-cert\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.402602 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.402363 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-trusted-ca-bundle\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.402602 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.402412 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-service-ca\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.402971 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.402937 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-config\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.403077 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.403058 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-oauth-serving-cert\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.403166 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.403148 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-service-ca\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.403303 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.403281 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-trusted-ca-bundle\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.404965 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.404942 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-oauth-config\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.405059 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.404942 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-serving-cert\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.409934 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.409913 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7p6q\" (UniqueName: \"kubernetes.io/projected/39bbee47-517e-4777-90b9-a6bb9be74ddf-kube-api-access-r7p6q\") pod \"console-fbb89dcd9-hfd9r\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.530551 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.530462 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:15:55.657495 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:55.657473 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fbb89dcd9-hfd9r"] Jul 23 00:15:55.659507 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:15:55.659475 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39bbee47_517e_4777_90b9_a6bb9be74ddf.slice/crio-a89afb9d918c9e244a760d02bd5253ad4b2edffd66bad0b3213a99115a6e4f14 WatchSource:0}: Error finding container a89afb9d918c9e244a760d02bd5253ad4b2edffd66bad0b3213a99115a6e4f14: Status 404 returned error can't find the container with id a89afb9d918c9e244a760d02bd5253ad4b2edffd66bad0b3213a99115a6e4f14 Jul 23 00:15:56.258427 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:56.258373 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbb89dcd9-hfd9r" event={"ID":"39bbee47-517e-4777-90b9-a6bb9be74ddf","Type":"ContainerStarted","Data":"d19c03392e3667048a671719d20aa446c900dad9fb6b4d29e1df94e5ea980359"} Jul 23 00:15:56.258827 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:56.258443 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbb89dcd9-hfd9r" event={"ID":"39bbee47-517e-4777-90b9-a6bb9be74ddf","Type":"ContainerStarted","Data":"a89afb9d918c9e244a760d02bd5253ad4b2edffd66bad0b3213a99115a6e4f14"} Jul 23 00:15:56.276870 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:15:56.276821 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fbb89dcd9-hfd9r" podStartSLOduration=1.276806341 podStartE2EDuration="1.276806341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:15:56.275356471 +0000 UTC m=+476.236095089" watchObservedRunningTime="2026-07-23 00:15:56.276806341 +0000 UTC m=+476.237544960" Jul 23 00:16:05.530758 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:05.530727 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:16:05.531139 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:05.530776 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:16:05.535436 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:05.535415 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:16:06.299278 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:06.299250 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:16:06.347136 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:06.347105 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-db9cdddf4-t9hmp"] Jul 23 00:16:06.451959 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:06.451932 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:06.552614 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:06.552545 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:06.653311 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:06.653281 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:13.491370 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:13.491332 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:22.283219 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:22.283185 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:22.381026 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:22.380996 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:31.375158 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.375110 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-db9cdddf4-t9hmp" podUID="b6b2e658-ec91-4e9e-93db-0f6f3eab90d1" containerName="console" containerID="cri-o://f7b75ca0c59cebf0ba8418c211d3ae4df56e518c70fd5a1bc54e98cc4efa7fd3" gracePeriod=15 Jul 23 00:16:31.620508 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.620484 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:16:31.710640 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.710604 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh8mc\" (UniqueName: \"kubernetes.io/projected/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-kube-api-access-kh8mc\") pod \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " Jul 23 00:16:31.710640 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.710643 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-service-ca\") pod \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " Jul 23 00:16:31.710853 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.710670 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-oauth-config\") pod \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " Jul 23 00:16:31.710853 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.710697 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-oauth-serving-cert\") pod \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " Jul 23 00:16:31.710853 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.710735 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-config\") pod \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " Jul 23 00:16:31.710853 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.710771 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-serving-cert\") pod \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " Jul 23 00:16:31.710853 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.710806 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-trusted-ca-bundle\") pod \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\" (UID: \"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1\") " Jul 23 00:16:31.711170 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.711136 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-service-ca" (OuterVolumeSpecName: "service-ca") pod "b6b2e658-ec91-4e9e-93db-0f6f3eab90d1" (UID: "b6b2e658-ec91-4e9e-93db-0f6f3eab90d1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:16:31.711310 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.711217 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-config" (OuterVolumeSpecName: "console-config") pod "b6b2e658-ec91-4e9e-93db-0f6f3eab90d1" (UID: "b6b2e658-ec91-4e9e-93db-0f6f3eab90d1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:16:31.711486 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.711467 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b6b2e658-ec91-4e9e-93db-0f6f3eab90d1" (UID: "b6b2e658-ec91-4e9e-93db-0f6f3eab90d1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:16:31.711573 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.711487 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b6b2e658-ec91-4e9e-93db-0f6f3eab90d1" (UID: "b6b2e658-ec91-4e9e-93db-0f6f3eab90d1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:16:31.712932 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.712902 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b6b2e658-ec91-4e9e-93db-0f6f3eab90d1" (UID: "b6b2e658-ec91-4e9e-93db-0f6f3eab90d1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:16:31.713016 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.712933 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b6b2e658-ec91-4e9e-93db-0f6f3eab90d1" (UID: "b6b2e658-ec91-4e9e-93db-0f6f3eab90d1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:16:31.713016 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.712960 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-kube-api-access-kh8mc" (OuterVolumeSpecName: "kube-api-access-kh8mc") pod "b6b2e658-ec91-4e9e-93db-0f6f3eab90d1" (UID: "b6b2e658-ec91-4e9e-93db-0f6f3eab90d1"). InnerVolumeSpecName "kube-api-access-kh8mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:16:31.812321 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.812285 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-serving-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:16:31.812321 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.812314 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-trusted-ca-bundle\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:16:31.812321 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.812323 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh8mc\" (UniqueName: \"kubernetes.io/projected/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-kube-api-access-kh8mc\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:16:31.812569 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.812333 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:16:31.812569 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.812342 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-oauth-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:16:31.812569 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.812351 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-oauth-serving-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:16:31.812569 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:31.812359 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1-console-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:16:32.398849 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:32.398814 2555 generic.go:356] "Generic (PLEG): container finished" podID="b6b2e658-ec91-4e9e-93db-0f6f3eab90d1" containerID="f7b75ca0c59cebf0ba8418c211d3ae4df56e518c70fd5a1bc54e98cc4efa7fd3" exitCode=0 Jul 23 00:16:32.399285 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:32.398859 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db9cdddf4-t9hmp" event={"ID":"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1","Type":"ContainerDied","Data":"f7b75ca0c59cebf0ba8418c211d3ae4df56e518c70fd5a1bc54e98cc4efa7fd3"} Jul 23 00:16:32.399285 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:32.398871 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db9cdddf4-t9hmp" Jul 23 00:16:32.399285 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:32.398929 2555 scope.go:117] "RemoveContainer" containerID="f7b75ca0c59cebf0ba8418c211d3ae4df56e518c70fd5a1bc54e98cc4efa7fd3" Jul 23 00:16:32.399285 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:32.398918 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db9cdddf4-t9hmp" event={"ID":"b6b2e658-ec91-4e9e-93db-0f6f3eab90d1","Type":"ContainerDied","Data":"1544632786fd28cc88c9b98fb99fd40a2836911aab3513d141363b31ac1a843b"} Jul 23 00:16:32.407415 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:32.407380 2555 scope.go:117] "RemoveContainer" containerID="f7b75ca0c59cebf0ba8418c211d3ae4df56e518c70fd5a1bc54e98cc4efa7fd3" Jul 23 00:16:32.407692 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:16:32.407676 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b75ca0c59cebf0ba8418c211d3ae4df56e518c70fd5a1bc54e98cc4efa7fd3\": container with ID starting with f7b75ca0c59cebf0ba8418c211d3ae4df56e518c70fd5a1bc54e98cc4efa7fd3 not found: ID does not exist" containerID="f7b75ca0c59cebf0ba8418c211d3ae4df56e518c70fd5a1bc54e98cc4efa7fd3" Jul 23 00:16:32.407744 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:32.407702 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b75ca0c59cebf0ba8418c211d3ae4df56e518c70fd5a1bc54e98cc4efa7fd3"} err="failed to get container status \"f7b75ca0c59cebf0ba8418c211d3ae4df56e518c70fd5a1bc54e98cc4efa7fd3\": rpc error: code = NotFound desc = could not find container \"f7b75ca0c59cebf0ba8418c211d3ae4df56e518c70fd5a1bc54e98cc4efa7fd3\": container with ID starting with f7b75ca0c59cebf0ba8418c211d3ae4df56e518c70fd5a1bc54e98cc4efa7fd3 not found: ID does not exist" Jul 23 00:16:32.418929 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:32.418903 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-db9cdddf4-t9hmp"] Jul 23 00:16:32.420680 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:32.420661 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-db9cdddf4-t9hmp"] Jul 23 00:16:32.501314 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:32.501279 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6b2e658-ec91-4e9e-93db-0f6f3eab90d1" path="/var/lib/kubelet/pods/b6b2e658-ec91-4e9e-93db-0f6f3eab90d1/volumes" Jul 23 00:16:46.400527 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:46.400491 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:48.943433 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:48.943400 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:49.741298 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:49.741263 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:52.569247 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:52.569156 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:52.662229 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:52.662196 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:53.957889 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:53.957832 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:54.058796 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.058764 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:54.136449 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.136420 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4p5ft"] Jul 23 00:16:54.136720 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.136709 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6b2e658-ec91-4e9e-93db-0f6f3eab90d1" containerName="console" Jul 23 00:16:54.136764 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.136722 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6b2e658-ec91-4e9e-93db-0f6f3eab90d1" containerName="console" Jul 23 00:16:54.136797 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.136776 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6b2e658-ec91-4e9e-93db-0f6f3eab90d1" containerName="console" Jul 23 00:16:54.139668 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.139652 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4p5ft" Jul 23 00:16:54.142217 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.142199 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 23 00:16:54.142931 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.142914 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 23 00:16:54.143031 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.142934 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hlqjm\"" Jul 23 00:16:54.151812 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.151791 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4p5ft"] Jul 23 00:16:54.303986 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.303896 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cd07224-1b66-4bf1-ae82-c2e8600f61bf-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4p5ft\" (UID: \"2cd07224-1b66-4bf1-ae82-c2e8600f61bf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4p5ft" Jul 23 00:16:54.303986 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.303946 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx8wj\" (UniqueName: \"kubernetes.io/projected/2cd07224-1b66-4bf1-ae82-c2e8600f61bf-kube-api-access-mx8wj\") pod \"cert-manager-webhook-6486ffd997-4p5ft\" (UID: \"2cd07224-1b66-4bf1-ae82-c2e8600f61bf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4p5ft" Jul 23 00:16:54.404414 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.404366 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cd07224-1b66-4bf1-ae82-c2e8600f61bf-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4p5ft\" (UID: \"2cd07224-1b66-4bf1-ae82-c2e8600f61bf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4p5ft" Jul 23 00:16:54.404620 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.404441 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx8wj\" (UniqueName: \"kubernetes.io/projected/2cd07224-1b66-4bf1-ae82-c2e8600f61bf-kube-api-access-mx8wj\") pod \"cert-manager-webhook-6486ffd997-4p5ft\" (UID: \"2cd07224-1b66-4bf1-ae82-c2e8600f61bf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4p5ft" Jul 23 00:16:54.405243 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.405222 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:54.415276 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.415245 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cd07224-1b66-4bf1-ae82-c2e8600f61bf-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4p5ft\" (UID: \"2cd07224-1b66-4bf1-ae82-c2e8600f61bf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4p5ft" Jul 23 00:16:54.415585 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.415568 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx8wj\" (UniqueName: \"kubernetes.io/projected/2cd07224-1b66-4bf1-ae82-c2e8600f61bf-kube-api-access-mx8wj\") pod \"cert-manager-webhook-6486ffd997-4p5ft\" (UID: \"2cd07224-1b66-4bf1-ae82-c2e8600f61bf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4p5ft" Jul 23 00:16:54.458552 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.458491 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4p5ft" Jul 23 00:16:54.509630 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.509509 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:54.602563 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:54.602534 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4p5ft"] Jul 23 00:16:54.604534 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:16:54.604508 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cd07224_1b66_4bf1_ae82_c2e8600f61bf.slice/crio-2ae4291c085507f3d78bea29b68ca8bc4576a50ff5d0067910c3e548edf0586b WatchSource:0}: Error finding container 2ae4291c085507f3d78bea29b68ca8bc4576a50ff5d0067910c3e548edf0586b: Status 404 returned error can't find the container with id 2ae4291c085507f3d78bea29b68ca8bc4576a50ff5d0067910c3e548edf0586b Jul 23 00:16:55.488843 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:55.488801 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4p5ft" event={"ID":"2cd07224-1b66-4bf1-ae82-c2e8600f61bf","Type":"ContainerStarted","Data":"2ae4291c085507f3d78bea29b68ca8bc4576a50ff5d0067910c3e548edf0586b"} Jul 23 00:16:55.961754 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:55.961721 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:56.354365 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:56.354216 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:58.504086 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:58.504053 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4p5ft" event={"ID":"2cd07224-1b66-4bf1-ae82-c2e8600f61bf","Type":"ContainerStarted","Data":"4369cb889b3ec89234cd69b658eb2d74d0ebba4f3c4df28935b2ad1e1400bc70"} Jul 23 00:16:58.504492 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:58.504112 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4p5ft" Jul 23 00:16:58.522503 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:58.522455 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-4p5ft" podStartSLOduration=1.139768685 podStartE2EDuration="4.522436118s" totalImagesPullingTime="3.382667433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:16:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:16:58.521605573 +0000 UTC m=+538.482344191" watchObservedRunningTime="2026-07-23 00:16:58.522436118 +0000 UTC m=+538.483174739" Jul 23 00:16:59.753527 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:59.753485 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:16:59.854291 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:16:59.854259 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:02.039504 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:02.039472 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:02.112298 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:02.112268 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:04.510821 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:04.510794 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4p5ft" Jul 23 00:17:04.769992 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:04.769918 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:05.374234 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:05.374201 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:05.471288 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:05.471252 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:06.074273 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:06.074237 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9jxhv"] Jul 23 00:17:06.078934 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:06.078914 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9jxhv" Jul 23 00:17:06.081178 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:06.081160 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gl2s9\"" Jul 23 00:17:06.087273 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:06.087247 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9jxhv"] Jul 23 00:17:06.213346 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:06.213312 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9d9a267-42fc-442c-9c5b-7724fb5c9509-bound-sa-token\") pod \"cert-manager-55b4d954d5-9jxhv\" (UID: \"c9d9a267-42fc-442c-9c5b-7724fb5c9509\") " pod="cert-manager/cert-manager-55b4d954d5-9jxhv" Jul 23 00:17:06.213552 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:06.213418 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fscbf\" (UniqueName: \"kubernetes.io/projected/c9d9a267-42fc-442c-9c5b-7724fb5c9509-kube-api-access-fscbf\") pod \"cert-manager-55b4d954d5-9jxhv\" (UID: \"c9d9a267-42fc-442c-9c5b-7724fb5c9509\") " pod="cert-manager/cert-manager-55b4d954d5-9jxhv" Jul 23 00:17:06.314731 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:06.314697 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9d9a267-42fc-442c-9c5b-7724fb5c9509-bound-sa-token\") pod \"cert-manager-55b4d954d5-9jxhv\" (UID: \"c9d9a267-42fc-442c-9c5b-7724fb5c9509\") " pod="cert-manager/cert-manager-55b4d954d5-9jxhv" Jul 23 00:17:06.314873 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:06.314751 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fscbf\" (UniqueName: \"kubernetes.io/projected/c9d9a267-42fc-442c-9c5b-7724fb5c9509-kube-api-access-fscbf\") pod \"cert-manager-55b4d954d5-9jxhv\" (UID: \"c9d9a267-42fc-442c-9c5b-7724fb5c9509\") " pod="cert-manager/cert-manager-55b4d954d5-9jxhv" Jul 23 00:17:06.323765 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:06.323726 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9d9a267-42fc-442c-9c5b-7724fb5c9509-bound-sa-token\") pod \"cert-manager-55b4d954d5-9jxhv\" (UID: \"c9d9a267-42fc-442c-9c5b-7724fb5c9509\") " pod="cert-manager/cert-manager-55b4d954d5-9jxhv" Jul 23 00:17:06.324013 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:06.323992 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fscbf\" (UniqueName: \"kubernetes.io/projected/c9d9a267-42fc-442c-9c5b-7724fb5c9509-kube-api-access-fscbf\") pod \"cert-manager-55b4d954d5-9jxhv\" (UID: \"c9d9a267-42fc-442c-9c5b-7724fb5c9509\") " pod="cert-manager/cert-manager-55b4d954d5-9jxhv" Jul 23 00:17:06.388882 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:06.388805 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9jxhv" Jul 23 00:17:06.517576 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:06.517555 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9jxhv"] Jul 23 00:17:06.519331 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:17:06.519297 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9d9a267_42fc_442c_9c5b_7724fb5c9509.slice/crio-b62f495da8b651ea085bca22d7f9f4e472116cf345c02538860476d72cb48d72 WatchSource:0}: Error finding container b62f495da8b651ea085bca22d7f9f4e472116cf345c02538860476d72cb48d72: Status 404 returned error can't find the container with id b62f495da8b651ea085bca22d7f9f4e472116cf345c02538860476d72cb48d72 Jul 23 00:17:06.539574 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:06.539549 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9jxhv" event={"ID":"c9d9a267-42fc-442c-9c5b-7724fb5c9509","Type":"ContainerStarted","Data":"b62f495da8b651ea085bca22d7f9f4e472116cf345c02538860476d72cb48d72"} Jul 23 00:17:07.544845 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:07.544808 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9jxhv" event={"ID":"c9d9a267-42fc-442c-9c5b-7724fb5c9509","Type":"ContainerStarted","Data":"287b9601c1bf015b473e4113824dd9e9dc0c8f5a3f2cccb5373e87a6a13408fa"} Jul 23 00:17:07.560357 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:07.560314 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-9jxhv" podStartSLOduration=1.5602987449999999 podStartE2EDuration="1.560298745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:17:07.559319829 +0000 UTC m=+547.520058448" watchObservedRunningTime="2026-07-23 00:17:07.560298745 +0000 UTC m=+547.521037364" Jul 23 00:17:11.678050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:11.678017 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:11.778913 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:11.778836 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:13.567481 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:13.567445 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:26.391503 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:26.391473 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:26.800498 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:26.800448 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:26.972250 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:26.972219 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:27.223568 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.223533 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:27.327282 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.327253 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:27.379764 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.379736 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2"] Jul 23 00:17:27.383192 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.383173 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:27.386234 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.386212 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 00:17:27.386338 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.386218 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mblxv\"" Jul 23 00:17:27.386488 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.386472 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 00:17:27.387036 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.387021 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 00:17:27.387147 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.387128 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 00:17:27.397074 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.397050 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2"] Jul 23 00:17:27.485069 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.484955 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:27.489110 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.489089 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/322c84c8-b4a2-4e9f-983a-84b63db49538-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2\" (UID: \"322c84c8-b4a2-4e9f-983a-84b63db49538\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:27.489197 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.489121 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc5r5\" (UniqueName: \"kubernetes.io/projected/322c84c8-b4a2-4e9f-983a-84b63db49538-kube-api-access-pc5r5\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2\" (UID: \"322c84c8-b4a2-4e9f-983a-84b63db49538\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:27.489197 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.489142 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/322c84c8-b4a2-4e9f-983a-84b63db49538-manifests\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2\" (UID: \"322c84c8-b4a2-4e9f-983a-84b63db49538\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:27.489197 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.489175 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/322c84c8-b4a2-4e9f-983a-84b63db49538-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2\" (UID: \"322c84c8-b4a2-4e9f-983a-84b63db49538\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:27.489350 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.489260 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/322c84c8-b4a2-4e9f-983a-84b63db49538-tmp\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2\" (UID: \"322c84c8-b4a2-4e9f-983a-84b63db49538\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:27.585842 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.585812 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:27.590623 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.590595 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/322c84c8-b4a2-4e9f-983a-84b63db49538-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2\" (UID: \"322c84c8-b4a2-4e9f-983a-84b63db49538\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:27.590764 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.590627 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc5r5\" (UniqueName: \"kubernetes.io/projected/322c84c8-b4a2-4e9f-983a-84b63db49538-kube-api-access-pc5r5\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2\" (UID: \"322c84c8-b4a2-4e9f-983a-84b63db49538\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:27.590764 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.590647 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/322c84c8-b4a2-4e9f-983a-84b63db49538-manifests\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2\" (UID: \"322c84c8-b4a2-4e9f-983a-84b63db49538\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:27.590764 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.590677 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/322c84c8-b4a2-4e9f-983a-84b63db49538-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2\" (UID: \"322c84c8-b4a2-4e9f-983a-84b63db49538\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:27.590764 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.590721 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/322c84c8-b4a2-4e9f-983a-84b63db49538-tmp\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2\" (UID: \"322c84c8-b4a2-4e9f-983a-84b63db49538\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:27.591187 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.591146 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/322c84c8-b4a2-4e9f-983a-84b63db49538-manifests\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2\" (UID: \"322c84c8-b4a2-4e9f-983a-84b63db49538\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:27.592972 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.592951 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/322c84c8-b4a2-4e9f-983a-84b63db49538-tmp\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2\" (UID: \"322c84c8-b4a2-4e9f-983a-84b63db49538\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:27.593083 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.593004 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/322c84c8-b4a2-4e9f-983a-84b63db49538-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2\" (UID: \"322c84c8-b4a2-4e9f-983a-84b63db49538\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:27.593175 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.593155 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/322c84c8-b4a2-4e9f-983a-84b63db49538-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2\" (UID: \"322c84c8-b4a2-4e9f-983a-84b63db49538\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:27.597969 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.597948 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc5r5\" (UniqueName: \"kubernetes.io/projected/322c84c8-b4a2-4e9f-983a-84b63db49538-kube-api-access-pc5r5\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2\" (UID: \"322c84c8-b4a2-4e9f-983a-84b63db49538\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:27.686864 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.686839 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:27.693988 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.693968 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:27.787096 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.787069 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:27.850247 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.850221 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2"] Jul 23 00:17:27.851774 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:17:27.851745 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322c84c8_b4a2_4e9f_983a_84b63db49538.slice/crio-b52b3cd72e5800022e7151e00874f5c07cba628a617f5a9f6e07bd2f49528aa6 WatchSource:0}: Error finding container b52b3cd72e5800022e7151e00874f5c07cba628a617f5a9f6e07bd2f49528aa6: Status 404 returned error can't find the container with id b52b3cd72e5800022e7151e00874f5c07cba628a617f5a9f6e07bd2f49528aa6 Jul 23 00:17:27.912506 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:27.912468 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:28.625162 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:28.625122 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" event={"ID":"322c84c8-b4a2-4e9f-983a-84b63db49538","Type":"ContainerStarted","Data":"b52b3cd72e5800022e7151e00874f5c07cba628a617f5a9f6e07bd2f49528aa6"} Jul 23 00:17:31.640857 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:31.640823 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" event={"ID":"322c84c8-b4a2-4e9f-983a-84b63db49538","Type":"ContainerStarted","Data":"c6db6038ea3c24051d5484912a863b23ab0fa96dbd4cd10d52f9e947399d42a4"} Jul 23 00:17:32.646032 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:32.646001 2555 generic.go:356] "Generic (PLEG): container finished" podID="322c84c8-b4a2-4e9f-983a-84b63db49538" containerID="c6db6038ea3c24051d5484912a863b23ab0fa96dbd4cd10d52f9e947399d42a4" exitCode=0 Jul 23 00:17:32.646032 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:32.646038 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" event={"ID":"322c84c8-b4a2-4e9f-983a-84b63db49538","Type":"ContainerDied","Data":"c6db6038ea3c24051d5484912a863b23ab0fa96dbd4cd10d52f9e947399d42a4"} Jul 23 00:17:33.652121 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:33.652090 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" event={"ID":"322c84c8-b4a2-4e9f-983a-84b63db49538","Type":"ContainerStarted","Data":"7ece3b4e8ab6b735bb5795d277d54ce2589a9ae2db55bf31a983776c13eec4f9"} Jul 23 00:17:33.652531 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:33.652196 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:33.675249 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:33.675201 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" podStartSLOduration=2.139420875 podStartE2EDuration="6.67518575s" totalImagesPullingTime="3.503867443s" totalInitContainerRuntime="1.031897432s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:17:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:17:33.673612716 +0000 UTC m=+573.634351336" watchObservedRunningTime="2026-07-23 00:17:33.67518575 +0000 UTC m=+573.635924369" Jul 23 00:17:33.858634 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:33.858602 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:35.164983 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:35.164951 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:38.548582 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:38.548550 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:38.648429 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:38.648401 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:44.658311 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:44.658272 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2" Jul 23 00:17:46.416317 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.416276 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:46.566442 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.566412 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g"] Jul 23 00:17:46.574734 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.574702 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" Jul 23 00:17:46.583265 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.583234 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 00:17:46.583603 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.583584 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 23 00:17:46.583708 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.583617 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 23 00:17:46.585971 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.585949 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:17:46.587106 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.587086 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 23 00:17:46.606493 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.606461 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-g4bpr\"" Jul 23 00:17:46.664161 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.664124 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff067220-60af-4687-b2e3-613d56520cfb-cert\") pod \"lws-controller-manager-85d467fcb7-86m4g\" (UID: \"ff067220-60af-4687-b2e3-613d56520cfb\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" Jul 23 00:17:46.664436 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.664186 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff067220-60af-4687-b2e3-613d56520cfb-metrics-cert\") pod \"lws-controller-manager-85d467fcb7-86m4g\" (UID: \"ff067220-60af-4687-b2e3-613d56520cfb\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" Jul 23 00:17:46.664436 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.664239 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ff067220-60af-4687-b2e3-613d56520cfb-manager-config\") pod \"lws-controller-manager-85d467fcb7-86m4g\" (UID: \"ff067220-60af-4687-b2e3-613d56520cfb\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" Jul 23 00:17:46.664436 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.664269 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mldsv\" (UniqueName: \"kubernetes.io/projected/ff067220-60af-4687-b2e3-613d56520cfb-kube-api-access-mldsv\") pod \"lws-controller-manager-85d467fcb7-86m4g\" (UID: \"ff067220-60af-4687-b2e3-613d56520cfb\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" Jul 23 00:17:46.726182 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.726107 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g"] Jul 23 00:17:46.738643 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.738614 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:46.764776 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.764740 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff067220-60af-4687-b2e3-613d56520cfb-cert\") pod \"lws-controller-manager-85d467fcb7-86m4g\" (UID: \"ff067220-60af-4687-b2e3-613d56520cfb\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" Jul 23 00:17:46.764954 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.764796 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff067220-60af-4687-b2e3-613d56520cfb-metrics-cert\") pod \"lws-controller-manager-85d467fcb7-86m4g\" (UID: \"ff067220-60af-4687-b2e3-613d56520cfb\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" Jul 23 00:17:46.764954 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.764824 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ff067220-60af-4687-b2e3-613d56520cfb-manager-config\") pod \"lws-controller-manager-85d467fcb7-86m4g\" (UID: \"ff067220-60af-4687-b2e3-613d56520cfb\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" Jul 23 00:17:46.764954 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.764844 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mldsv\" (UniqueName: \"kubernetes.io/projected/ff067220-60af-4687-b2e3-613d56520cfb-kube-api-access-mldsv\") pod \"lws-controller-manager-85d467fcb7-86m4g\" (UID: \"ff067220-60af-4687-b2e3-613d56520cfb\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" Jul 23 00:17:46.765545 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.765521 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ff067220-60af-4687-b2e3-613d56520cfb-manager-config\") pod \"lws-controller-manager-85d467fcb7-86m4g\" (UID: \"ff067220-60af-4687-b2e3-613d56520cfb\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" Jul 23 00:17:46.767286 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.767264 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff067220-60af-4687-b2e3-613d56520cfb-cert\") pod \"lws-controller-manager-85d467fcb7-86m4g\" (UID: \"ff067220-60af-4687-b2e3-613d56520cfb\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" Jul 23 00:17:46.767366 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.767300 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff067220-60af-4687-b2e3-613d56520cfb-metrics-cert\") pod \"lws-controller-manager-85d467fcb7-86m4g\" (UID: \"ff067220-60af-4687-b2e3-613d56520cfb\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" Jul 23 00:17:46.778219 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.778195 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mldsv\" (UniqueName: \"kubernetes.io/projected/ff067220-60af-4687-b2e3-613d56520cfb-kube-api-access-mldsv\") pod \"lws-controller-manager-85d467fcb7-86m4g\" (UID: \"ff067220-60af-4687-b2e3-613d56520cfb\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" Jul 23 00:17:46.888940 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.888908 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" Jul 23 00:17:46.892911 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:46.892887 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:47.035497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.035460 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g"] Jul 23 00:17:47.037688 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:17:47.037642 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff067220_60af_4687_b2e3_613d56520cfb.slice/crio-97912fb7e3996ffb08194142756892e95ef5daf83c2d0c90eea85077e5b7f1e5 WatchSource:0}: Error finding container 97912fb7e3996ffb08194142756892e95ef5daf83c2d0c90eea85077e5b7f1e5: Status 404 returned error can't find the container with id 97912fb7e3996ffb08194142756892e95ef5daf83c2d0c90eea85077e5b7f1e5 Jul 23 00:17:47.226569 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.226540 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7447dc7bcf-qlpg6"] Jul 23 00:17:47.231340 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.231262 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-qlpg6" Jul 23 00:17:47.233892 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.233867 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 00:17:47.234010 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.233876 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-62c26\"" Jul 23 00:17:47.234146 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.234131 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 23 00:17:47.234212 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.234193 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 00:17:47.234299 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.234281 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 23 00:17:47.245641 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.245616 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7447dc7bcf-qlpg6"] Jul 23 00:17:47.300121 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.300073 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:47.370541 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.370508 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a01cbd-db3e-4851-8306-0db08ea7e428-tls-certs\") pod \"kube-auth-proxy-7447dc7bcf-qlpg6\" (UID: \"76a01cbd-db3e-4851-8306-0db08ea7e428\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-qlpg6" Jul 23 00:17:47.370541 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.370544 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76a01cbd-db3e-4851-8306-0db08ea7e428-tmp\") pod \"kube-auth-proxy-7447dc7bcf-qlpg6\" (UID: \"76a01cbd-db3e-4851-8306-0db08ea7e428\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-qlpg6" Jul 23 00:17:47.370801 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.370669 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvxcb\" (UniqueName: \"kubernetes.io/projected/76a01cbd-db3e-4851-8306-0db08ea7e428-kube-api-access-cvxcb\") pod \"kube-auth-proxy-7447dc7bcf-qlpg6\" (UID: \"76a01cbd-db3e-4851-8306-0db08ea7e428\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-qlpg6" Jul 23 00:17:47.387569 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.387535 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:47.471890 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.471850 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a01cbd-db3e-4851-8306-0db08ea7e428-tls-certs\") pod \"kube-auth-proxy-7447dc7bcf-qlpg6\" (UID: \"76a01cbd-db3e-4851-8306-0db08ea7e428\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-qlpg6" Jul 23 00:17:47.472443 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.471905 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76a01cbd-db3e-4851-8306-0db08ea7e428-tmp\") pod \"kube-auth-proxy-7447dc7bcf-qlpg6\" (UID: \"76a01cbd-db3e-4851-8306-0db08ea7e428\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-qlpg6" Jul 23 00:17:47.472443 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.472013 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvxcb\" (UniqueName: \"kubernetes.io/projected/76a01cbd-db3e-4851-8306-0db08ea7e428-kube-api-access-cvxcb\") pod \"kube-auth-proxy-7447dc7bcf-qlpg6\" (UID: \"76a01cbd-db3e-4851-8306-0db08ea7e428\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-qlpg6" Jul 23 00:17:47.474269 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.474242 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76a01cbd-db3e-4851-8306-0db08ea7e428-tmp\") pod \"kube-auth-proxy-7447dc7bcf-qlpg6\" (UID: \"76a01cbd-db3e-4851-8306-0db08ea7e428\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-qlpg6" Jul 23 00:17:47.474504 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.474486 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a01cbd-db3e-4851-8306-0db08ea7e428-tls-certs\") pod \"kube-auth-proxy-7447dc7bcf-qlpg6\" (UID: \"76a01cbd-db3e-4851-8306-0db08ea7e428\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-qlpg6" Jul 23 00:17:47.479327 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.479301 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvxcb\" (UniqueName: \"kubernetes.io/projected/76a01cbd-db3e-4851-8306-0db08ea7e428-kube-api-access-cvxcb\") pod \"kube-auth-proxy-7447dc7bcf-qlpg6\" (UID: \"76a01cbd-db3e-4851-8306-0db08ea7e428\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-qlpg6" Jul 23 00:17:47.487425 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.487345 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:47.542548 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.542512 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-qlpg6" Jul 23 00:17:47.686997 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.686584 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7447dc7bcf-qlpg6"] Jul 23 00:17:47.689349 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:17:47.689320 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a01cbd_db3e_4851_8306_0db08ea7e428.slice/crio-448b7c076160823a88e8cdbcda618a367e2812ac0c0facd9f7db826767bc7317 WatchSource:0}: Error finding container 448b7c076160823a88e8cdbcda618a367e2812ac0c0facd9f7db826767bc7317: Status 404 returned error can't find the container with id 448b7c076160823a88e8cdbcda618a367e2812ac0c0facd9f7db826767bc7317 Jul 23 00:17:47.714481 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.714439 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-qlpg6" event={"ID":"76a01cbd-db3e-4851-8306-0db08ea7e428","Type":"ContainerStarted","Data":"448b7c076160823a88e8cdbcda618a367e2812ac0c0facd9f7db826767bc7317"} Jul 23 00:17:47.716517 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.716426 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" event={"ID":"ff067220-60af-4687-b2e3-613d56520cfb","Type":"ContainerStarted","Data":"97912fb7e3996ffb08194142756892e95ef5daf83c2d0c90eea85077e5b7f1e5"} Jul 23 00:17:47.736748 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:47.736666 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:48.200707 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:48.200676 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:48.615599 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:48.615485 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:48.672523 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:48.672484 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:48.810212 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:48.810171 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:49.484060 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:49.484025 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:49.730099 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:49.730056 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" event={"ID":"ff067220-60af-4687-b2e3-613d56520cfb","Type":"ContainerStarted","Data":"42b83f730216a850557fc3dadfc442a5fc097ab3fe93f29cbcb8aa8db2181465"} Jul 23 00:17:49.730492 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:49.730135 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" Jul 23 00:17:49.745266 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:49.745169 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" podStartSLOduration=1.204660291 podStartE2EDuration="3.745150729s" totalImagesPullingTime="2.540490438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:17:49.744512535 +0000 UTC m=+589.705251154" watchObservedRunningTime="2026-07-23 00:17:49.745150729 +0000 UTC m=+589.705889340" Jul 23 00:17:50.736782 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:50.736747 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-qlpg6" event={"ID":"76a01cbd-db3e-4851-8306-0db08ea7e428","Type":"ContainerStarted","Data":"ebb0af8bf963315c8e4e50f0cc5a98564729a3b7a265c5a0ff43c695076a93a0"} Jul 23 00:17:50.755175 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:50.754897 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-qlpg6" podStartSLOduration=1.066991902 podStartE2EDuration="3.75487585s" totalImagesPullingTime="2.687883948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:17:50.753028609 +0000 UTC m=+590.713767241" watchObservedRunningTime="2026-07-23 00:17:50.75487585 +0000 UTC m=+590.715614470" Jul 23 00:17:53.935072 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:53.935029 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:54.040658 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:54.040622 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:54.688766 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:54.688731 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:54.931797 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:54.931767 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:55.043784 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:55.043699 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:57.480639 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:57.480586 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:57.571481 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:57.571350 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:57.828709 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:57.828627 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:57.991631 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:57.991593 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:58.763537 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:58.763505 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:58.864139 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:58.864109 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:17:58.963622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:17:58.963591 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:18:00.393230 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:18:00.393201 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:18:00.394042 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:18:00.394023 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:18:00.739086 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:18:00.739063 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-86m4g" Jul 23 00:18:05.888553 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:18:05.888520 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:18:05.988922 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:18:05.988893 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:18:24.397593 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:18:24.397518 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:18:26.344264 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:18:26.344232 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:18:26.398341 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:18:26.398296 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:18:47.987064 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:18:47.987030 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:18:48.086714 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:18:48.086676 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:18:48.168072 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:18:48.168043 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:17.297603 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:17.297566 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:20.686487 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:20.686452 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:21.465443 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:21.465413 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:25.087964 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:25.087932 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:25.316305 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:25.316271 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:27.369649 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:27.369607 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:27.981148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:27.981118 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:28.363830 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:28.363754 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:30.336522 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.336480 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:30.372634 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.372605 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d64c7cf85-l4nl5"] Jul 23 00:19:30.376259 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.376238 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.386031 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.386004 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d64c7cf85-l4nl5"] Jul 23 00:19:30.434917 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.434878 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:30.438113 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.438086 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a71497f-0c0d-46ec-ab70-2805745ea5c4-trusted-ca-bundle\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.438253 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.438123 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a71497f-0c0d-46ec-ab70-2805745ea5c4-oauth-serving-cert\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.438253 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.438183 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a71497f-0c0d-46ec-ab70-2805745ea5c4-console-config\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.438253 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.438226 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a71497f-0c0d-46ec-ab70-2805745ea5c4-console-oauth-config\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.438374 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.438266 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a71497f-0c0d-46ec-ab70-2805745ea5c4-service-ca\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.438374 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.438343 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkjj7\" (UniqueName: \"kubernetes.io/projected/4a71497f-0c0d-46ec-ab70-2805745ea5c4-kube-api-access-hkjj7\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.438374 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.438366 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a71497f-0c0d-46ec-ab70-2805745ea5c4-console-serving-cert\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.535764 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.535734 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:30.539398 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.539368 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkjj7\" (UniqueName: \"kubernetes.io/projected/4a71497f-0c0d-46ec-ab70-2805745ea5c4-kube-api-access-hkjj7\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.539513 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.539422 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a71497f-0c0d-46ec-ab70-2805745ea5c4-console-serving-cert\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.539513 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.539469 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a71497f-0c0d-46ec-ab70-2805745ea5c4-trusted-ca-bundle\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.539513 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.539488 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a71497f-0c0d-46ec-ab70-2805745ea5c4-oauth-serving-cert\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.539668 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.539519 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a71497f-0c0d-46ec-ab70-2805745ea5c4-console-config\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.539668 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.539562 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a71497f-0c0d-46ec-ab70-2805745ea5c4-console-oauth-config\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.539668 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.539595 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a71497f-0c0d-46ec-ab70-2805745ea5c4-service-ca\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.540428 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.540405 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a71497f-0c0d-46ec-ab70-2805745ea5c4-oauth-serving-cert\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.540558 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.540436 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a71497f-0c0d-46ec-ab70-2805745ea5c4-console-config\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.540558 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.540453 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a71497f-0c0d-46ec-ab70-2805745ea5c4-service-ca\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.540558 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.540480 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a71497f-0c0d-46ec-ab70-2805745ea5c4-trusted-ca-bundle\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.542126 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.542101 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a71497f-0c0d-46ec-ab70-2805745ea5c4-console-oauth-config\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.542218 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.542178 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a71497f-0c0d-46ec-ab70-2805745ea5c4-console-serving-cert\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.551835 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.551810 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkjj7\" (UniqueName: \"kubernetes.io/projected/4a71497f-0c0d-46ec-ab70-2805745ea5c4-kube-api-access-hkjj7\") pod \"console-6d64c7cf85-l4nl5\" (UID: \"4a71497f-0c0d-46ec-ab70-2805745ea5c4\") " pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.636065 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.635979 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:30.686832 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.686800 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:30.823698 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:30.823674 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d64c7cf85-l4nl5"] Jul 23 00:19:30.826295 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:19:30.826268 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a71497f_0c0d_46ec_ab70_2805745ea5c4.slice/crio-823b506ad7a21875a9663e49927bc2426bc5b88dc0156d7f3ef5676df9143420 WatchSource:0}: Error finding container 823b506ad7a21875a9663e49927bc2426bc5b88dc0156d7f3ef5676df9143420: Status 404 returned error can't find the container with id 823b506ad7a21875a9663e49927bc2426bc5b88dc0156d7f3ef5676df9143420 Jul 23 00:19:31.149158 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:31.149127 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:31.183873 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:31.183838 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d64c7cf85-l4nl5" event={"ID":"4a71497f-0c0d-46ec-ab70-2805745ea5c4","Type":"ContainerStarted","Data":"6e9cc458c777ff2a46d6376c7212bc9cf3045213c9338305ff9369ef7bb87948"} Jul 23 00:19:31.184036 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:31.183892 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d64c7cf85-l4nl5" event={"ID":"4a71497f-0c0d-46ec-ab70-2805745ea5c4","Type":"ContainerStarted","Data":"823b506ad7a21875a9663e49927bc2426bc5b88dc0156d7f3ef5676df9143420"} Jul 23 00:19:31.200600 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:31.200555 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d64c7cf85-l4nl5" podStartSLOduration=1.200539118 podStartE2EDuration="1.200539118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:19:31.199427909 +0000 UTC m=+691.160166526" watchObservedRunningTime="2026-07-23 00:19:31.200539118 +0000 UTC m=+691.161277738" Jul 23 00:19:31.250690 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:31.250653 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:33.297671 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:33.297639 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:33.371612 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:33.371585 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:38.687656 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:38.687591 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:38.794253 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:38.794209 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:39.136094 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:39.136007 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:39.460237 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:39.460209 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:40.116847 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:40.116817 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:40.306243 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:40.306209 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:40.687530 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:40.687495 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:40.687688 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:40.687558 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:40.692231 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:40.692207 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:41.232700 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:41.232672 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d64c7cf85-l4nl5" Jul 23 00:19:41.278583 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:41.278550 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fbb89dcd9-hfd9r"] Jul 23 00:19:41.437194 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:41.437160 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:41.532470 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:41.532380 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:42.881901 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:42.881870 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:43.265307 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:43.265275 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:45.023545 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:45.023512 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:45.124076 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:45.124051 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:51.213543 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:51.213511 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggwgh"] Jul 23 00:19:51.225304 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:51.225268 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggwgh" Jul 23 00:19:51.226366 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:51.226340 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggwgh"] Jul 23 00:19:51.227772 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:51.227744 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 00:19:51.228552 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:51.228530 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8zc48\"" Jul 23 00:19:51.228676 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:51.228538 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 00:19:51.326985 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:51.326949 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pqmg\" (UniqueName: \"kubernetes.io/projected/f748a5da-aa5e-4904-8c16-abe0232e1045-kube-api-access-6pqmg\") pod \"limitador-operator-controller-manager-859d7b78bd-ggwgh\" (UID: \"f748a5da-aa5e-4904-8c16-abe0232e1045\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggwgh" Jul 23 00:19:51.428405 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:51.428340 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pqmg\" (UniqueName: \"kubernetes.io/projected/f748a5da-aa5e-4904-8c16-abe0232e1045-kube-api-access-6pqmg\") pod \"limitador-operator-controller-manager-859d7b78bd-ggwgh\" (UID: \"f748a5da-aa5e-4904-8c16-abe0232e1045\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggwgh" Jul 23 00:19:51.437888 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:51.437867 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pqmg\" (UniqueName: \"kubernetes.io/projected/f748a5da-aa5e-4904-8c16-abe0232e1045-kube-api-access-6pqmg\") pod \"limitador-operator-controller-manager-859d7b78bd-ggwgh\" (UID: \"f748a5da-aa5e-4904-8c16-abe0232e1045\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggwgh" Jul 23 00:19:51.550885 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:51.550792 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggwgh" Jul 23 00:19:51.691329 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:51.691300 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggwgh"] Jul 23 00:19:51.693251 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:19:51.693225 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf748a5da_aa5e_4904_8c16_abe0232e1045.slice/crio-99eef40bcc22c3e646304ec8ca35de4db98ae854773a1bd144786c09c3d8f564 WatchSource:0}: Error finding container 99eef40bcc22c3e646304ec8ca35de4db98ae854773a1bd144786c09c3d8f564: Status 404 returned error can't find the container with id 99eef40bcc22c3e646304ec8ca35de4db98ae854773a1bd144786c09c3d8f564 Jul 23 00:19:51.695309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:51.695290 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:19:51.696211 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:51.696184 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:51.802000 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:51.801926 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:52.278152 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:52.278122 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggwgh" event={"ID":"f748a5da-aa5e-4904-8c16-abe0232e1045","Type":"ContainerStarted","Data":"99eef40bcc22c3e646304ec8ca35de4db98ae854773a1bd144786c09c3d8f564"} Jul 23 00:19:54.293723 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:54.293689 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggwgh" event={"ID":"f748a5da-aa5e-4904-8c16-abe0232e1045","Type":"ContainerStarted","Data":"9f1d4a1054cfc8d93d2090e31b358b69a266ff9e4261e22c4cf7470cffe35d42"} Jul 23 00:19:54.294115 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:54.293758 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggwgh" Jul 23 00:19:54.312337 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:54.312283 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggwgh" podStartSLOduration=1.589450257 podStartE2EDuration="3.312270603s" totalImagesPullingTime="1.722820346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:19:54.310738415 +0000 UTC m=+714.271477033" watchObservedRunningTime="2026-07-23 00:19:54.312270603 +0000 UTC m=+714.273009220" Jul 23 00:19:57.049646 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:57.049614 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:57.151831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:57.151801 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:57.648980 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:57.648954 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:19:57.704613 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:19:57.704588 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:01.047037 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:01.047004 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:01.149492 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:01.149457 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:04.491568 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:04.491538 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:04.592586 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:04.592542 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:05.299972 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:05.299943 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggwgh" Jul 23 00:20:06.300125 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.300080 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-fbb89dcd9-hfd9r" podUID="39bbee47-517e-4777-90b9-a6bb9be74ddf" containerName="console" containerID="cri-o://d19c03392e3667048a671719d20aa446c900dad9fb6b4d29e1df94e5ea980359" gracePeriod=15 Jul 23 00:20:06.550647 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.550595 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:20:06.660796 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.660758 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7p6q\" (UniqueName: \"kubernetes.io/projected/39bbee47-517e-4777-90b9-a6bb9be74ddf-kube-api-access-r7p6q\") pod \"39bbee47-517e-4777-90b9-a6bb9be74ddf\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " Jul 23 00:20:06.660796 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.660793 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-oauth-config\") pod \"39bbee47-517e-4777-90b9-a6bb9be74ddf\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " Jul 23 00:20:06.661032 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.660832 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-config\") pod \"39bbee47-517e-4777-90b9-a6bb9be74ddf\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " Jul 23 00:20:06.661032 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.660854 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-service-ca\") pod \"39bbee47-517e-4777-90b9-a6bb9be74ddf\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " Jul 23 00:20:06.661032 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.660882 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-oauth-serving-cert\") pod \"39bbee47-517e-4777-90b9-a6bb9be74ddf\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " Jul 23 00:20:06.661032 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.660926 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-serving-cert\") pod \"39bbee47-517e-4777-90b9-a6bb9be74ddf\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " Jul 23 00:20:06.661032 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.660950 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-trusted-ca-bundle\") pod \"39bbee47-517e-4777-90b9-a6bb9be74ddf\" (UID: \"39bbee47-517e-4777-90b9-a6bb9be74ddf\") " Jul 23 00:20:06.661257 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.661224 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-config" (OuterVolumeSpecName: "console-config") pod "39bbee47-517e-4777-90b9-a6bb9be74ddf" (UID: "39bbee47-517e-4777-90b9-a6bb9be74ddf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:20:06.661447 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.661416 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "39bbee47-517e-4777-90b9-a6bb9be74ddf" (UID: "39bbee47-517e-4777-90b9-a6bb9be74ddf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:20:06.661583 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.661513 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-service-ca" (OuterVolumeSpecName: "service-ca") pod "39bbee47-517e-4777-90b9-a6bb9be74ddf" (UID: "39bbee47-517e-4777-90b9-a6bb9be74ddf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:20:06.661583 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.661554 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "39bbee47-517e-4777-90b9-a6bb9be74ddf" (UID: "39bbee47-517e-4777-90b9-a6bb9be74ddf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:20:06.663124 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.663088 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "39bbee47-517e-4777-90b9-a6bb9be74ddf" (UID: "39bbee47-517e-4777-90b9-a6bb9be74ddf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:20:06.663124 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.663100 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39bbee47-517e-4777-90b9-a6bb9be74ddf-kube-api-access-r7p6q" (OuterVolumeSpecName: "kube-api-access-r7p6q") pod "39bbee47-517e-4777-90b9-a6bb9be74ddf" (UID: "39bbee47-517e-4777-90b9-a6bb9be74ddf"). InnerVolumeSpecName "kube-api-access-r7p6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:20:06.663269 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.663102 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "39bbee47-517e-4777-90b9-a6bb9be74ddf" (UID: "39bbee47-517e-4777-90b9-a6bb9be74ddf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:20:06.761590 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.761558 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:20:06.761590 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.761583 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:20:06.761590 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.761592 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-oauth-serving-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:20:06.761590 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.761602 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-serving-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:20:06.761848 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.761611 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39bbee47-517e-4777-90b9-a6bb9be74ddf-trusted-ca-bundle\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:20:06.761848 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.761619 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7p6q\" (UniqueName: \"kubernetes.io/projected/39bbee47-517e-4777-90b9-a6bb9be74ddf-kube-api-access-r7p6q\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:20:06.761848 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:06.761628 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39bbee47-517e-4777-90b9-a6bb9be74ddf-console-oauth-config\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:20:07.354650 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:07.354614 2555 generic.go:356] "Generic (PLEG): container finished" podID="39bbee47-517e-4777-90b9-a6bb9be74ddf" containerID="d19c03392e3667048a671719d20aa446c900dad9fb6b4d29e1df94e5ea980359" exitCode=0 Jul 23 00:20:07.354650 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:07.354658 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbb89dcd9-hfd9r" Jul 23 00:20:07.355086 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:07.354670 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbb89dcd9-hfd9r" event={"ID":"39bbee47-517e-4777-90b9-a6bb9be74ddf","Type":"ContainerDied","Data":"d19c03392e3667048a671719d20aa446c900dad9fb6b4d29e1df94e5ea980359"} Jul 23 00:20:07.355086 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:07.354714 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbb89dcd9-hfd9r" event={"ID":"39bbee47-517e-4777-90b9-a6bb9be74ddf","Type":"ContainerDied","Data":"a89afb9d918c9e244a760d02bd5253ad4b2edffd66bad0b3213a99115a6e4f14"} Jul 23 00:20:07.355086 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:07.354742 2555 scope.go:117] "RemoveContainer" containerID="d19c03392e3667048a671719d20aa446c900dad9fb6b4d29e1df94e5ea980359" Jul 23 00:20:07.363599 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:07.363583 2555 scope.go:117] "RemoveContainer" containerID="d19c03392e3667048a671719d20aa446c900dad9fb6b4d29e1df94e5ea980359" Jul 23 00:20:07.363868 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:20:07.363850 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19c03392e3667048a671719d20aa446c900dad9fb6b4d29e1df94e5ea980359\": container with ID starting with d19c03392e3667048a671719d20aa446c900dad9fb6b4d29e1df94e5ea980359 not found: ID does not exist" containerID="d19c03392e3667048a671719d20aa446c900dad9fb6b4d29e1df94e5ea980359" Jul 23 00:20:07.363921 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:07.363879 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19c03392e3667048a671719d20aa446c900dad9fb6b4d29e1df94e5ea980359"} err="failed to get container status \"d19c03392e3667048a671719d20aa446c900dad9fb6b4d29e1df94e5ea980359\": rpc error: code = NotFound desc = could not find container \"d19c03392e3667048a671719d20aa446c900dad9fb6b4d29e1df94e5ea980359\": container with ID starting with d19c03392e3667048a671719d20aa446c900dad9fb6b4d29e1df94e5ea980359 not found: ID does not exist" Jul 23 00:20:07.375251 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:07.375228 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fbb89dcd9-hfd9r"] Jul 23 00:20:07.380620 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:07.380598 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fbb89dcd9-hfd9r"] Jul 23 00:20:08.501806 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:08.501766 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39bbee47-517e-4777-90b9-a6bb9be74ddf" path="/var/lib/kubelet/pods/39bbee47-517e-4777-90b9-a6bb9be74ddf/volumes" Jul 23 00:20:22.779054 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:22.779022 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:22.830703 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:22.830671 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:27.553708 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:27.553674 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:27.610087 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:27.610057 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:33.038569 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:33.038531 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:33.524882 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:33.524847 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:44.067951 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:44.067919 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:44.173598 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:44.173555 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:44.274197 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:44.274166 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:44.870521 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:44.870487 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-qmg9f"] Jul 23 00:20:44.870835 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:44.870822 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39bbee47-517e-4777-90b9-a6bb9be74ddf" containerName="console" Jul 23 00:20:44.870879 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:44.870837 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="39bbee47-517e-4777-90b9-a6bb9be74ddf" containerName="console" Jul 23 00:20:44.870929 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:44.870919 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="39bbee47-517e-4777-90b9-a6bb9be74ddf" containerName="console" Jul 23 00:20:44.873900 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:44.873883 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qmg9f" Jul 23 00:20:44.876064 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:44.876046 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ksmj2\"" Jul 23 00:20:44.879948 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:44.879922 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qmg9f"] Jul 23 00:20:44.899628 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:44.899602 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c64cq\" (UniqueName: \"kubernetes.io/projected/b8c14627-5bd7-4f3f-a1c0-9ed7a5563960-kube-api-access-c64cq\") pod \"authorino-58cd955675-qmg9f\" (UID: \"b8c14627-5bd7-4f3f-a1c0-9ed7a5563960\") " pod="kuadrant-system/authorino-58cd955675-qmg9f" Jul 23 00:20:44.905940 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:44.905919 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:45.000229 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:45.000185 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c64cq\" (UniqueName: \"kubernetes.io/projected/b8c14627-5bd7-4f3f-a1c0-9ed7a5563960-kube-api-access-c64cq\") pod \"authorino-58cd955675-qmg9f\" (UID: \"b8c14627-5bd7-4f3f-a1c0-9ed7a5563960\") " pod="kuadrant-system/authorino-58cd955675-qmg9f" Jul 23 00:20:45.008899 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:45.008869 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-hxrnk"] Jul 23 00:20:45.010950 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:45.010922 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c64cq\" (UniqueName: \"kubernetes.io/projected/b8c14627-5bd7-4f3f-a1c0-9ed7a5563960-kube-api-access-c64cq\") pod \"authorino-58cd955675-qmg9f\" (UID: \"b8c14627-5bd7-4f3f-a1c0-9ed7a5563960\") " pod="kuadrant-system/authorino-58cd955675-qmg9f" Jul 23 00:20:45.012458 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:45.012441 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hxrnk" Jul 23 00:20:45.018432 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:45.018413 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hxrnk"] Jul 23 00:20:45.045052 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:45.045014 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:20:45.100948 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:45.100913 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9prz\" (UniqueName: \"kubernetes.io/projected/120f09b7-be13-4731-991d-5b6b8839b5ad-kube-api-access-v9prz\") pod \"authorino-75d556bd77-hxrnk\" (UID: \"120f09b7-be13-4731-991d-5b6b8839b5ad\") " pod="kuadrant-system/authorino-75d556bd77-hxrnk" Jul 23 00:20:45.184465 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:45.184439 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qmg9f" Jul 23 00:20:45.201800 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:45.201776 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9prz\" (UniqueName: \"kubernetes.io/projected/120f09b7-be13-4731-991d-5b6b8839b5ad-kube-api-access-v9prz\") pod \"authorino-75d556bd77-hxrnk\" (UID: \"120f09b7-be13-4731-991d-5b6b8839b5ad\") " pod="kuadrant-system/authorino-75d556bd77-hxrnk" Jul 23 00:20:45.209621 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:45.209595 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9prz\" (UniqueName: \"kubernetes.io/projected/120f09b7-be13-4731-991d-5b6b8839b5ad-kube-api-access-v9prz\") pod \"authorino-75d556bd77-hxrnk\" (UID: \"120f09b7-be13-4731-991d-5b6b8839b5ad\") " pod="kuadrant-system/authorino-75d556bd77-hxrnk" Jul 23 00:20:45.313802 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:45.313770 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qmg9f"] Jul 23 00:20:45.317043 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:20:45.317015 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8c14627_5bd7_4f3f_a1c0_9ed7a5563960.slice/crio-8d9d4a53b7796e59c28270990874cff16c8d1c0451081d2b8b0cfe6f7640c796 WatchSource:0}: Error finding container 8d9d4a53b7796e59c28270990874cff16c8d1c0451081d2b8b0cfe6f7640c796: Status 404 returned error can't find the container with id 8d9d4a53b7796e59c28270990874cff16c8d1c0451081d2b8b0cfe6f7640c796 Jul 23 00:20:45.326784 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:45.326767 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hxrnk" Jul 23 00:20:45.451605 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:45.451576 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hxrnk"] Jul 23 00:20:45.453932 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:20:45.453909 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod120f09b7_be13_4731_991d_5b6b8839b5ad.slice/crio-dbe64312bd0ddc54cf8f426a54d1f4f4f93283b08c112a47a8cea3b7558cc92e WatchSource:0}: Error finding container dbe64312bd0ddc54cf8f426a54d1f4f4f93283b08c112a47a8cea3b7558cc92e: Status 404 returned error can't find the container with id dbe64312bd0ddc54cf8f426a54d1f4f4f93283b08c112a47a8cea3b7558cc92e Jul 23 00:20:45.522181 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:45.522147 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hxrnk" event={"ID":"120f09b7-be13-4731-991d-5b6b8839b5ad","Type":"ContainerStarted","Data":"dbe64312bd0ddc54cf8f426a54d1f4f4f93283b08c112a47a8cea3b7558cc92e"} Jul 23 00:20:45.523233 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:45.523202 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qmg9f" event={"ID":"b8c14627-5bd7-4f3f-a1c0-9ed7a5563960","Type":"ContainerStarted","Data":"8d9d4a53b7796e59c28270990874cff16c8d1c0451081d2b8b0cfe6f7640c796"} Jul 23 00:20:49.545609 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:49.545576 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hxrnk" event={"ID":"120f09b7-be13-4731-991d-5b6b8839b5ad","Type":"ContainerStarted","Data":"ffaadac6c2cc8960b11f9074b71413fbdf5942df44d53a96c7b7a25fe375e61e"} Jul 23 00:20:49.547113 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:49.547086 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qmg9f" event={"ID":"b8c14627-5bd7-4f3f-a1c0-9ed7a5563960","Type":"ContainerStarted","Data":"89c93ae2dbb5a2f520e86dd44bcc9d0c90665d95fd0b3ba56dae952374adaf67"} Jul 23 00:20:49.562063 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:49.562018 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-hxrnk" podStartSLOduration=2.447710168 podStartE2EDuration="5.562004947s" totalImagesPullingTime="3.114294779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:49.560519338 +0000 UTC m=+769.521257958" watchObservedRunningTime="2026-07-23 00:20:49.562004947 +0000 UTC m=+769.522743565" Jul 23 00:20:49.575226 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:49.575181 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-qmg9f" podStartSLOduration=2.335600659 podStartE2EDuration="5.575164316s" totalImagesPullingTime="3.239563657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:49.573948119 +0000 UTC m=+769.534686739" watchObservedRunningTime="2026-07-23 00:20:49.575164316 +0000 UTC m=+769.535902934" Jul 23 00:20:49.601209 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:49.601178 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qmg9f"] Jul 23 00:20:51.559318 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:51.559281 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-qmg9f" podUID="b8c14627-5bd7-4f3f-a1c0-9ed7a5563960" containerName="authorino" containerID="cri-o://89c93ae2dbb5a2f520e86dd44bcc9d0c90665d95fd0b3ba56dae952374adaf67" gracePeriod=30 Jul 23 00:20:51.811675 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:51.811601 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qmg9f" Jul 23 00:20:51.858021 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:51.857991 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c64cq\" (UniqueName: \"kubernetes.io/projected/b8c14627-5bd7-4f3f-a1c0-9ed7a5563960-kube-api-access-c64cq\") pod \"b8c14627-5bd7-4f3f-a1c0-9ed7a5563960\" (UID: \"b8c14627-5bd7-4f3f-a1c0-9ed7a5563960\") " Jul 23 00:20:51.860101 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:51.860079 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c14627-5bd7-4f3f-a1c0-9ed7a5563960-kube-api-access-c64cq" (OuterVolumeSpecName: "kube-api-access-c64cq") pod "b8c14627-5bd7-4f3f-a1c0-9ed7a5563960" (UID: "b8c14627-5bd7-4f3f-a1c0-9ed7a5563960"). InnerVolumeSpecName "kube-api-access-c64cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:20:51.959053 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:51.959007 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c64cq\" (UniqueName: \"kubernetes.io/projected/b8c14627-5bd7-4f3f-a1c0-9ed7a5563960-kube-api-access-c64cq\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:20:52.564724 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:52.564685 2555 generic.go:356] "Generic (PLEG): container finished" podID="b8c14627-5bd7-4f3f-a1c0-9ed7a5563960" containerID="89c93ae2dbb5a2f520e86dd44bcc9d0c90665d95fd0b3ba56dae952374adaf67" exitCode=0 Jul 23 00:20:52.564724 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:52.564726 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qmg9f" Jul 23 00:20:52.565248 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:52.564757 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qmg9f" event={"ID":"b8c14627-5bd7-4f3f-a1c0-9ed7a5563960","Type":"ContainerDied","Data":"89c93ae2dbb5a2f520e86dd44bcc9d0c90665d95fd0b3ba56dae952374adaf67"} Jul 23 00:20:52.565248 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:52.564795 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qmg9f" event={"ID":"b8c14627-5bd7-4f3f-a1c0-9ed7a5563960","Type":"ContainerDied","Data":"8d9d4a53b7796e59c28270990874cff16c8d1c0451081d2b8b0cfe6f7640c796"} Jul 23 00:20:52.565248 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:52.564817 2555 scope.go:117] "RemoveContainer" containerID="89c93ae2dbb5a2f520e86dd44bcc9d0c90665d95fd0b3ba56dae952374adaf67" Jul 23 00:20:52.573191 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:52.573168 2555 scope.go:117] "RemoveContainer" containerID="89c93ae2dbb5a2f520e86dd44bcc9d0c90665d95fd0b3ba56dae952374adaf67" Jul 23 00:20:52.573456 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:20:52.573435 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c93ae2dbb5a2f520e86dd44bcc9d0c90665d95fd0b3ba56dae952374adaf67\": container with ID starting with 89c93ae2dbb5a2f520e86dd44bcc9d0c90665d95fd0b3ba56dae952374adaf67 not found: ID does not exist" containerID="89c93ae2dbb5a2f520e86dd44bcc9d0c90665d95fd0b3ba56dae952374adaf67" Jul 23 00:20:52.573528 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:52.573465 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c93ae2dbb5a2f520e86dd44bcc9d0c90665d95fd0b3ba56dae952374adaf67"} err="failed to get container status \"89c93ae2dbb5a2f520e86dd44bcc9d0c90665d95fd0b3ba56dae952374adaf67\": rpc error: code = NotFound desc = could not find container \"89c93ae2dbb5a2f520e86dd44bcc9d0c90665d95fd0b3ba56dae952374adaf67\": container with ID starting with 89c93ae2dbb5a2f520e86dd44bcc9d0c90665d95fd0b3ba56dae952374adaf67 not found: ID does not exist" Jul 23 00:20:52.580646 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:52.580617 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qmg9f"] Jul 23 00:20:52.585609 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:52.585574 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-qmg9f"] Jul 23 00:20:54.502182 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:20:54.502149 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8c14627-5bd7-4f3f-a1c0-9ed7a5563960" path="/var/lib/kubelet/pods/b8c14627-5bd7-4f3f-a1c0-9ed7a5563960/volumes" Jul 23 00:21:04.126129 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:04.126086 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:04.271079 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:04.271048 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:04.630687 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:04.630654 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:16.679018 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:16.678988 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:16.912746 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:16.912715 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-69zc2"] Jul 23 00:21:16.913055 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:16.913043 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c14627-5bd7-4f3f-a1c0-9ed7a5563960" containerName="authorino" Jul 23 00:21:16.913102 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:16.913056 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c14627-5bd7-4f3f-a1c0-9ed7a5563960" containerName="authorino" Jul 23 00:21:16.913137 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:16.913128 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c14627-5bd7-4f3f-a1c0-9ed7a5563960" containerName="authorino" Jul 23 00:21:16.917411 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:16.917380 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-69zc2" Jul 23 00:21:16.922287 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:16.922265 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-69zc2"] Jul 23 00:21:17.079720 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.079644 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws6h7\" (UniqueName: \"kubernetes.io/projected/b1115ff2-35db-4e91-918d-9e5f01ef2966-kube-api-access-ws6h7\") pod \"authorino-5d4b67c446-69zc2\" (UID: \"b1115ff2-35db-4e91-918d-9e5f01ef2966\") " pod="kuadrant-system/authorino-5d4b67c446-69zc2" Jul 23 00:21:17.163846 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.163811 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-69zc2"] Jul 23 00:21:17.164092 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:21:17.164069 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ws6h7], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-69zc2" podUID="b1115ff2-35db-4e91-918d-9e5f01ef2966" Jul 23 00:21:17.180703 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.180671 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws6h7\" (UniqueName: \"kubernetes.io/projected/b1115ff2-35db-4e91-918d-9e5f01ef2966-kube-api-access-ws6h7\") pod \"authorino-5d4b67c446-69zc2\" (UID: \"b1115ff2-35db-4e91-918d-9e5f01ef2966\") " pod="kuadrant-system/authorino-5d4b67c446-69zc2" Jul 23 00:21:17.183154 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.183131 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c55f9598-df2mn"] Jul 23 00:21:17.186582 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.186563 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c55f9598-df2mn" Jul 23 00:21:17.191385 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.191364 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws6h7\" (UniqueName: \"kubernetes.io/projected/b1115ff2-35db-4e91-918d-9e5f01ef2966-kube-api-access-ws6h7\") pod \"authorino-5d4b67c446-69zc2\" (UID: \"b1115ff2-35db-4e91-918d-9e5f01ef2966\") " pod="kuadrant-system/authorino-5d4b67c446-69zc2" Jul 23 00:21:17.192821 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.192797 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c55f9598-df2mn"] Jul 23 00:21:17.195644 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.195616 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:17.281998 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.281964 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pbnk\" (UniqueName: \"kubernetes.io/projected/751e24c0-188c-42a0-ab07-7bad683f2726-kube-api-access-2pbnk\") pod \"authorino-5c55f9598-df2mn\" (UID: \"751e24c0-188c-42a0-ab07-7bad683f2726\") " pod="kuadrant-system/authorino-5c55f9598-df2mn" Jul 23 00:21:17.383468 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.383370 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pbnk\" (UniqueName: \"kubernetes.io/projected/751e24c0-188c-42a0-ab07-7bad683f2726-kube-api-access-2pbnk\") pod \"authorino-5c55f9598-df2mn\" (UID: \"751e24c0-188c-42a0-ab07-7bad683f2726\") " pod="kuadrant-system/authorino-5c55f9598-df2mn" Jul 23 00:21:17.391784 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.391756 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pbnk\" (UniqueName: \"kubernetes.io/projected/751e24c0-188c-42a0-ab07-7bad683f2726-kube-api-access-2pbnk\") pod \"authorino-5c55f9598-df2mn\" (UID: \"751e24c0-188c-42a0-ab07-7bad683f2726\") " pod="kuadrant-system/authorino-5c55f9598-df2mn" Jul 23 00:21:17.451194 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.451162 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c55f9598-df2mn"] Jul 23 00:21:17.451445 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.451433 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c55f9598-df2mn" Jul 23 00:21:17.480003 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.479977 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7558cfdcd-bnm8d"] Jul 23 00:21:17.483534 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.483519 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7558cfdcd-bnm8d" Jul 23 00:21:17.485778 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.485759 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 23 00:21:17.490747 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.490725 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7558cfdcd-bnm8d"] Jul 23 00:21:17.579124 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.579101 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c55f9598-df2mn"] Jul 23 00:21:17.581081 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:21:17.581055 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod751e24c0_188c_42a0_ab07_7bad683f2726.slice/crio-4b78661b78312e8be1f75ecd6474137411e4687a98682f1e5b233430387dad4c WatchSource:0}: Error finding container 4b78661b78312e8be1f75ecd6474137411e4687a98682f1e5b233430387dad4c: Status 404 returned error can't find the container with id 4b78661b78312e8be1f75ecd6474137411e4687a98682f1e5b233430387dad4c Jul 23 00:21:17.585314 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.585291 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e6fbb509-6676-44ad-8809-432f8566fce7-tls-cert\") pod \"authorino-7558cfdcd-bnm8d\" (UID: \"e6fbb509-6676-44ad-8809-432f8566fce7\") " pod="kuadrant-system/authorino-7558cfdcd-bnm8d" Jul 23 00:21:17.585415 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.585333 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58knp\" (UniqueName: \"kubernetes.io/projected/e6fbb509-6676-44ad-8809-432f8566fce7-kube-api-access-58knp\") pod \"authorino-7558cfdcd-bnm8d\" (UID: \"e6fbb509-6676-44ad-8809-432f8566fce7\") " pod="kuadrant-system/authorino-7558cfdcd-bnm8d" Jul 23 00:21:17.671203 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.671177 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c55f9598-df2mn" event={"ID":"751e24c0-188c-42a0-ab07-7bad683f2726","Type":"ContainerStarted","Data":"4b78661b78312e8be1f75ecd6474137411e4687a98682f1e5b233430387dad4c"} Jul 23 00:21:17.671203 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.671202 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-69zc2" Jul 23 00:21:17.678184 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.678164 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-69zc2" Jul 23 00:21:17.686050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.686026 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e6fbb509-6676-44ad-8809-432f8566fce7-tls-cert\") pod \"authorino-7558cfdcd-bnm8d\" (UID: \"e6fbb509-6676-44ad-8809-432f8566fce7\") " pod="kuadrant-system/authorino-7558cfdcd-bnm8d" Jul 23 00:21:17.686373 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.686060 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58knp\" (UniqueName: \"kubernetes.io/projected/e6fbb509-6676-44ad-8809-432f8566fce7-kube-api-access-58knp\") pod \"authorino-7558cfdcd-bnm8d\" (UID: \"e6fbb509-6676-44ad-8809-432f8566fce7\") " pod="kuadrant-system/authorino-7558cfdcd-bnm8d" Jul 23 00:21:17.688365 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.688348 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e6fbb509-6676-44ad-8809-432f8566fce7-tls-cert\") pod \"authorino-7558cfdcd-bnm8d\" (UID: \"e6fbb509-6676-44ad-8809-432f8566fce7\") " pod="kuadrant-system/authorino-7558cfdcd-bnm8d" Jul 23 00:21:17.693461 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.693440 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58knp\" (UniqueName: \"kubernetes.io/projected/e6fbb509-6676-44ad-8809-432f8566fce7-kube-api-access-58knp\") pod \"authorino-7558cfdcd-bnm8d\" (UID: \"e6fbb509-6676-44ad-8809-432f8566fce7\") " pod="kuadrant-system/authorino-7558cfdcd-bnm8d" Jul 23 00:21:17.787144 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.787117 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ws6h7\" (UniqueName: \"kubernetes.io/projected/b1115ff2-35db-4e91-918d-9e5f01ef2966-kube-api-access-ws6h7\") pod \"b1115ff2-35db-4e91-918d-9e5f01ef2966\" (UID: \"b1115ff2-35db-4e91-918d-9e5f01ef2966\") " Jul 23 00:21:17.788879 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.788849 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1115ff2-35db-4e91-918d-9e5f01ef2966-kube-api-access-ws6h7" (OuterVolumeSpecName: "kube-api-access-ws6h7") pod "b1115ff2-35db-4e91-918d-9e5f01ef2966" (UID: "b1115ff2-35db-4e91-918d-9e5f01ef2966"). InnerVolumeSpecName "kube-api-access-ws6h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:21:17.795438 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.795423 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7558cfdcd-bnm8d" Jul 23 00:21:17.888065 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.888035 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws6h7\" (UniqueName: \"kubernetes.io/projected/b1115ff2-35db-4e91-918d-9e5f01ef2966-kube-api-access-ws6h7\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:21:17.918749 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:17.918725 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7558cfdcd-bnm8d"] Jul 23 00:21:17.920649 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:21:17.920625 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6fbb509_6676_44ad_8809_432f8566fce7.slice/crio-7938168bc496fe8b75529204328f5feb814b1c125a9d072dab2229fedff030e0 WatchSource:0}: Error finding container 7938168bc496fe8b75529204328f5feb814b1c125a9d072dab2229fedff030e0: Status 404 returned error can't find the container with id 7938168bc496fe8b75529204328f5feb814b1c125a9d072dab2229fedff030e0 Jul 23 00:21:18.677225 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:18.677191 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c55f9598-df2mn" event={"ID":"751e24c0-188c-42a0-ab07-7bad683f2726","Type":"ContainerStarted","Data":"7defa72a3b26750107f5f30290ded5d129833020fe717718417d11ab9ed3b791"} Jul 23 00:21:18.677419 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:18.677243 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5c55f9598-df2mn" podUID="751e24c0-188c-42a0-ab07-7bad683f2726" containerName="authorino" containerID="cri-o://7defa72a3b26750107f5f30290ded5d129833020fe717718417d11ab9ed3b791" gracePeriod=30 Jul 23 00:21:18.678925 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:18.678893 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7558cfdcd-bnm8d" event={"ID":"e6fbb509-6676-44ad-8809-432f8566fce7","Type":"ContainerStarted","Data":"f5a7c1bd6c85324ce0bb7fbbcd079c7c4c3c768c98f0d76b92516b8908aae71c"} Jul 23 00:21:18.679062 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:18.678940 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7558cfdcd-bnm8d" event={"ID":"e6fbb509-6676-44ad-8809-432f8566fce7","Type":"ContainerStarted","Data":"7938168bc496fe8b75529204328f5feb814b1c125a9d072dab2229fedff030e0"} Jul 23 00:21:18.679062 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:18.678905 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-69zc2" Jul 23 00:21:18.692446 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:18.692402 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5c55f9598-df2mn" podStartSLOduration=0.961174546 podStartE2EDuration="1.69237159s" totalImagesPullingTime="731.197044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:21:18.69147798 +0000 UTC m=+798.652216600" watchObservedRunningTime="2026-07-23 00:21:18.69237159 +0000 UTC m=+798.653110208" Jul 23 00:21:18.712494 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:18.712467 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-69zc2"] Jul 23 00:21:18.716431 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:18.716407 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-69zc2"] Jul 23 00:21:18.753331 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:18.753267 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7558cfdcd-bnm8d" podStartSLOduration=1.344362541 podStartE2EDuration="1.753241626s" totalImagesPullingTime="408.879085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:21:18.728609733 +0000 UTC m=+798.689348353" watchObservedRunningTime="2026-07-23 00:21:18.753241626 +0000 UTC m=+798.713980247" Jul 23 00:21:18.753818 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:18.753799 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hxrnk"] Jul 23 00:21:18.754020 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:18.753997 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-hxrnk" podUID="120f09b7-be13-4731-991d-5b6b8839b5ad" containerName="authorino" containerID="cri-o://ffaadac6c2cc8960b11f9074b71413fbdf5942df44d53a96c7b7a25fe375e61e" gracePeriod=30 Jul 23 00:21:19.008621 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.008595 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c55f9598-df2mn" Jul 23 00:21:19.018381 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.018361 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hxrnk" Jul 23 00:21:19.097634 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.097593 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pbnk\" (UniqueName: \"kubernetes.io/projected/751e24c0-188c-42a0-ab07-7bad683f2726-kube-api-access-2pbnk\") pod \"751e24c0-188c-42a0-ab07-7bad683f2726\" (UID: \"751e24c0-188c-42a0-ab07-7bad683f2726\") " Jul 23 00:21:19.097803 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.097685 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9prz\" (UniqueName: \"kubernetes.io/projected/120f09b7-be13-4731-991d-5b6b8839b5ad-kube-api-access-v9prz\") pod \"120f09b7-be13-4731-991d-5b6b8839b5ad\" (UID: \"120f09b7-be13-4731-991d-5b6b8839b5ad\") " Jul 23 00:21:19.099622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.099591 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/120f09b7-be13-4731-991d-5b6b8839b5ad-kube-api-access-v9prz" (OuterVolumeSpecName: "kube-api-access-v9prz") pod "120f09b7-be13-4731-991d-5b6b8839b5ad" (UID: "120f09b7-be13-4731-991d-5b6b8839b5ad"). InnerVolumeSpecName "kube-api-access-v9prz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:21:19.099622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.099616 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751e24c0-188c-42a0-ab07-7bad683f2726-kube-api-access-2pbnk" (OuterVolumeSpecName: "kube-api-access-2pbnk") pod "751e24c0-188c-42a0-ab07-7bad683f2726" (UID: "751e24c0-188c-42a0-ab07-7bad683f2726"). InnerVolumeSpecName "kube-api-access-2pbnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:21:19.198422 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.198373 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9prz\" (UniqueName: \"kubernetes.io/projected/120f09b7-be13-4731-991d-5b6b8839b5ad-kube-api-access-v9prz\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:21:19.198574 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.198427 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pbnk\" (UniqueName: \"kubernetes.io/projected/751e24c0-188c-42a0-ab07-7bad683f2726-kube-api-access-2pbnk\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:21:19.327788 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.327707 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:19.685004 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.684971 2555 generic.go:356] "Generic (PLEG): container finished" podID="120f09b7-be13-4731-991d-5b6b8839b5ad" containerID="ffaadac6c2cc8960b11f9074b71413fbdf5942df44d53a96c7b7a25fe375e61e" exitCode=0 Jul 23 00:21:19.685194 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.685012 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hxrnk" Jul 23 00:21:19.685194 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.685054 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hxrnk" event={"ID":"120f09b7-be13-4731-991d-5b6b8839b5ad","Type":"ContainerDied","Data":"ffaadac6c2cc8960b11f9074b71413fbdf5942df44d53a96c7b7a25fe375e61e"} Jul 23 00:21:19.685194 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.685096 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hxrnk" event={"ID":"120f09b7-be13-4731-991d-5b6b8839b5ad","Type":"ContainerDied","Data":"dbe64312bd0ddc54cf8f426a54d1f4f4f93283b08c112a47a8cea3b7558cc92e"} Jul 23 00:21:19.685194 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.685117 2555 scope.go:117] "RemoveContainer" containerID="ffaadac6c2cc8960b11f9074b71413fbdf5942df44d53a96c7b7a25fe375e61e" Jul 23 00:21:19.686649 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.686311 2555 generic.go:356] "Generic (PLEG): container finished" podID="751e24c0-188c-42a0-ab07-7bad683f2726" containerID="7defa72a3b26750107f5f30290ded5d129833020fe717718417d11ab9ed3b791" exitCode=0 Jul 23 00:21:19.686649 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.686354 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c55f9598-df2mn" Jul 23 00:21:19.686649 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.686381 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c55f9598-df2mn" event={"ID":"751e24c0-188c-42a0-ab07-7bad683f2726","Type":"ContainerDied","Data":"7defa72a3b26750107f5f30290ded5d129833020fe717718417d11ab9ed3b791"} Jul 23 00:21:19.686649 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.686440 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c55f9598-df2mn" event={"ID":"751e24c0-188c-42a0-ab07-7bad683f2726","Type":"ContainerDied","Data":"4b78661b78312e8be1f75ecd6474137411e4687a98682f1e5b233430387dad4c"} Jul 23 00:21:19.693803 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.693777 2555 scope.go:117] "RemoveContainer" containerID="ffaadac6c2cc8960b11f9074b71413fbdf5942df44d53a96c7b7a25fe375e61e" Jul 23 00:21:19.694363 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:21:19.694047 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffaadac6c2cc8960b11f9074b71413fbdf5942df44d53a96c7b7a25fe375e61e\": container with ID starting with ffaadac6c2cc8960b11f9074b71413fbdf5942df44d53a96c7b7a25fe375e61e not found: ID does not exist" containerID="ffaadac6c2cc8960b11f9074b71413fbdf5942df44d53a96c7b7a25fe375e61e" Jul 23 00:21:19.694363 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.694081 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffaadac6c2cc8960b11f9074b71413fbdf5942df44d53a96c7b7a25fe375e61e"} err="failed to get container status \"ffaadac6c2cc8960b11f9074b71413fbdf5942df44d53a96c7b7a25fe375e61e\": rpc error: code = NotFound desc = could not find container \"ffaadac6c2cc8960b11f9074b71413fbdf5942df44d53a96c7b7a25fe375e61e\": container with ID starting with ffaadac6c2cc8960b11f9074b71413fbdf5942df44d53a96c7b7a25fe375e61e not found: ID does not exist" Jul 23 00:21:19.694363 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.694103 2555 scope.go:117] "RemoveContainer" containerID="7defa72a3b26750107f5f30290ded5d129833020fe717718417d11ab9ed3b791" Jul 23 00:21:19.701548 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.701533 2555 scope.go:117] "RemoveContainer" containerID="7defa72a3b26750107f5f30290ded5d129833020fe717718417d11ab9ed3b791" Jul 23 00:21:19.701792 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:21:19.701777 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7defa72a3b26750107f5f30290ded5d129833020fe717718417d11ab9ed3b791\": container with ID starting with 7defa72a3b26750107f5f30290ded5d129833020fe717718417d11ab9ed3b791 not found: ID does not exist" containerID="7defa72a3b26750107f5f30290ded5d129833020fe717718417d11ab9ed3b791" Jul 23 00:21:19.701833 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.701799 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7defa72a3b26750107f5f30290ded5d129833020fe717718417d11ab9ed3b791"} err="failed to get container status \"7defa72a3b26750107f5f30290ded5d129833020fe717718417d11ab9ed3b791\": rpc error: code = NotFound desc = could not find container \"7defa72a3b26750107f5f30290ded5d129833020fe717718417d11ab9ed3b791\": container with ID starting with 7defa72a3b26750107f5f30290ded5d129833020fe717718417d11ab9ed3b791 not found: ID does not exist" Jul 23 00:21:19.708150 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.708128 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hxrnk"] Jul 23 00:21:19.714075 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.714058 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hxrnk"] Jul 23 00:21:19.723310 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.723291 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c55f9598-df2mn"] Jul 23 00:21:19.729538 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:19.729518 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5c55f9598-df2mn"] Jul 23 00:21:20.502285 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:20.502253 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="120f09b7-be13-4731-991d-5b6b8839b5ad" path="/var/lib/kubelet/pods/120f09b7-be13-4731-991d-5b6b8839b5ad/volumes" Jul 23 00:21:20.502620 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:20.502605 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="751e24c0-188c-42a0-ab07-7bad683f2726" path="/var/lib/kubelet/pods/751e24c0-188c-42a0-ab07-7bad683f2726/volumes" Jul 23 00:21:20.502873 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:20.502862 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1115ff2-35db-4e91-918d-9e5f01ef2966" path="/var/lib/kubelet/pods/b1115ff2-35db-4e91-918d-9e5f01ef2966/volumes" Jul 23 00:21:23.555173 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.555095 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:23.574333 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.574297 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-69c97bc454-4g54d"] Jul 23 00:21:23.574647 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.574635 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="751e24c0-188c-42a0-ab07-7bad683f2726" containerName="authorino" Jul 23 00:21:23.574695 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.574649 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="751e24c0-188c-42a0-ab07-7bad683f2726" containerName="authorino" Jul 23 00:21:23.574695 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.574661 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="120f09b7-be13-4731-991d-5b6b8839b5ad" containerName="authorino" Jul 23 00:21:23.574695 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.574666 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="120f09b7-be13-4731-991d-5b6b8839b5ad" containerName="authorino" Jul 23 00:21:23.574789 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.574732 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="751e24c0-188c-42a0-ab07-7bad683f2726" containerName="authorino" Jul 23 00:21:23.574789 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.574742 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="120f09b7-be13-4731-991d-5b6b8839b5ad" containerName="authorino" Jul 23 00:21:23.580425 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.580403 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69c97bc454-4g54d" Jul 23 00:21:23.584203 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.584169 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4t6cw\"" Jul 23 00:21:23.584342 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.584216 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 23 00:21:23.584342 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.584221 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 23 00:21:23.587824 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.587750 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69c97bc454-4g54d"] Jul 23 00:21:23.704084 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.704050 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:23.739348 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.739312 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cf33b38-60f5-4625-8503-8ee8561f4ff8-webhook-cert\") pod \"maas-controller-69c97bc454-4g54d\" (UID: \"3cf33b38-60f5-4625-8503-8ee8561f4ff8\") " pod="opendatahub/maas-controller-69c97bc454-4g54d" Jul 23 00:21:23.739554 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.739358 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnlxc\" (UniqueName: \"kubernetes.io/projected/3cf33b38-60f5-4625-8503-8ee8561f4ff8-kube-api-access-jnlxc\") pod \"maas-controller-69c97bc454-4g54d\" (UID: \"3cf33b38-60f5-4625-8503-8ee8561f4ff8\") " pod="opendatahub/maas-controller-69c97bc454-4g54d" Jul 23 00:21:23.775011 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.774985 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:23.840177 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.840092 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cf33b38-60f5-4625-8503-8ee8561f4ff8-webhook-cert\") pod \"maas-controller-69c97bc454-4g54d\" (UID: \"3cf33b38-60f5-4625-8503-8ee8561f4ff8\") " pod="opendatahub/maas-controller-69c97bc454-4g54d" Jul 23 00:21:23.840177 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.840135 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnlxc\" (UniqueName: \"kubernetes.io/projected/3cf33b38-60f5-4625-8503-8ee8561f4ff8-kube-api-access-jnlxc\") pod \"maas-controller-69c97bc454-4g54d\" (UID: \"3cf33b38-60f5-4625-8503-8ee8561f4ff8\") " pod="opendatahub/maas-controller-69c97bc454-4g54d" Jul 23 00:21:23.842519 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.842496 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cf33b38-60f5-4625-8503-8ee8561f4ff8-webhook-cert\") pod \"maas-controller-69c97bc454-4g54d\" (UID: \"3cf33b38-60f5-4625-8503-8ee8561f4ff8\") " pod="opendatahub/maas-controller-69c97bc454-4g54d" Jul 23 00:21:23.847895 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.847873 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnlxc\" (UniqueName: \"kubernetes.io/projected/3cf33b38-60f5-4625-8503-8ee8561f4ff8-kube-api-access-jnlxc\") pod \"maas-controller-69c97bc454-4g54d\" (UID: \"3cf33b38-60f5-4625-8503-8ee8561f4ff8\") " pod="opendatahub/maas-controller-69c97bc454-4g54d" Jul 23 00:21:23.894204 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:23.894176 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69c97bc454-4g54d" Jul 23 00:21:24.023323 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:24.023296 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69c97bc454-4g54d"] Jul 23 00:21:24.025864 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:21:24.025831 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf33b38_60f5_4625_8503_8ee8561f4ff8.slice/crio-3dc727ba08665fda17a619120a9ac9f41cd07fe7b6fd0428aba1ee1b35e56b5e WatchSource:0}: Error finding container 3dc727ba08665fda17a619120a9ac9f41cd07fe7b6fd0428aba1ee1b35e56b5e: Status 404 returned error can't find the container with id 3dc727ba08665fda17a619120a9ac9f41cd07fe7b6fd0428aba1ee1b35e56b5e Jul 23 00:21:24.712971 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:24.712929 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69c97bc454-4g54d" event={"ID":"3cf33b38-60f5-4625-8503-8ee8561f4ff8","Type":"ContainerStarted","Data":"3dc727ba08665fda17a619120a9ac9f41cd07fe7b6fd0428aba1ee1b35e56b5e"} Jul 23 00:21:26.554162 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:26.554130 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:26.723918 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:26.723885 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69c97bc454-4g54d" event={"ID":"3cf33b38-60f5-4625-8503-8ee8561f4ff8","Type":"ContainerStarted","Data":"a26b7c32d1be7c9d421bfdce725d558560ac9cec2f23be4fe8cf94e5d0870243"} Jul 23 00:21:26.724093 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:26.723948 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-69c97bc454-4g54d" Jul 23 00:21:26.743542 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:26.743477 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-69c97bc454-4g54d" podStartSLOduration=1.383838795 podStartE2EDuration="3.743460951s" totalImagesPullingTime="2.359622156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:21:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:21:26.741674128 +0000 UTC m=+806.702412758" watchObservedRunningTime="2026-07-23 00:21:26.743460951 +0000 UTC m=+806.704199570" Jul 23 00:21:26.891193 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:26.891168 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:37.731341 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:37.731306 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-69c97bc454-4g54d" Jul 23 00:21:37.944505 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:37.944473 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69c97bc454-4g54d"] Jul 23 00:21:37.944699 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:37.944678 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-69c97bc454-4g54d" podUID="3cf33b38-60f5-4625-8503-8ee8561f4ff8" containerName="manager" containerID="cri-o://a26b7c32d1be7c9d421bfdce725d558560ac9cec2f23be4fe8cf94e5d0870243" gracePeriod=30 Jul 23 00:21:38.189959 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:38.189935 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69c97bc454-4g54d" Jul 23 00:21:38.264901 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:38.264873 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cf33b38-60f5-4625-8503-8ee8561f4ff8-webhook-cert\") pod \"3cf33b38-60f5-4625-8503-8ee8561f4ff8\" (UID: \"3cf33b38-60f5-4625-8503-8ee8561f4ff8\") " Jul 23 00:21:38.265050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:38.264929 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnlxc\" (UniqueName: \"kubernetes.io/projected/3cf33b38-60f5-4625-8503-8ee8561f4ff8-kube-api-access-jnlxc\") pod \"3cf33b38-60f5-4625-8503-8ee8561f4ff8\" (UID: \"3cf33b38-60f5-4625-8503-8ee8561f4ff8\") " Jul 23 00:21:38.267110 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:38.267078 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cf33b38-60f5-4625-8503-8ee8561f4ff8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3cf33b38-60f5-4625-8503-8ee8561f4ff8" (UID: "3cf33b38-60f5-4625-8503-8ee8561f4ff8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:21:38.267227 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:38.267178 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf33b38-60f5-4625-8503-8ee8561f4ff8-kube-api-access-jnlxc" (OuterVolumeSpecName: "kube-api-access-jnlxc") pod "3cf33b38-60f5-4625-8503-8ee8561f4ff8" (UID: "3cf33b38-60f5-4625-8503-8ee8561f4ff8"). InnerVolumeSpecName "kube-api-access-jnlxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:21:38.366163 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:38.366083 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnlxc\" (UniqueName: \"kubernetes.io/projected/3cf33b38-60f5-4625-8503-8ee8561f4ff8-kube-api-access-jnlxc\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:21:38.366163 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:38.366118 2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cf33b38-60f5-4625-8503-8ee8561f4ff8-webhook-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:21:38.782637 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:38.782603 2555 generic.go:356] "Generic (PLEG): container finished" podID="3cf33b38-60f5-4625-8503-8ee8561f4ff8" containerID="a26b7c32d1be7c9d421bfdce725d558560ac9cec2f23be4fe8cf94e5d0870243" exitCode=0 Jul 23 00:21:38.783068 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:38.782648 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69c97bc454-4g54d" Jul 23 00:21:38.783068 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:38.782690 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69c97bc454-4g54d" event={"ID":"3cf33b38-60f5-4625-8503-8ee8561f4ff8","Type":"ContainerDied","Data":"a26b7c32d1be7c9d421bfdce725d558560ac9cec2f23be4fe8cf94e5d0870243"} Jul 23 00:21:38.783068 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:38.782738 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69c97bc454-4g54d" event={"ID":"3cf33b38-60f5-4625-8503-8ee8561f4ff8","Type":"ContainerDied","Data":"3dc727ba08665fda17a619120a9ac9f41cd07fe7b6fd0428aba1ee1b35e56b5e"} Jul 23 00:21:38.783068 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:38.782764 2555 scope.go:117] "RemoveContainer" containerID="a26b7c32d1be7c9d421bfdce725d558560ac9cec2f23be4fe8cf94e5d0870243" Jul 23 00:21:38.790851 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:38.790836 2555 scope.go:117] "RemoveContainer" containerID="a26b7c32d1be7c9d421bfdce725d558560ac9cec2f23be4fe8cf94e5d0870243" Jul 23 00:21:38.791099 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:21:38.791079 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a26b7c32d1be7c9d421bfdce725d558560ac9cec2f23be4fe8cf94e5d0870243\": container with ID starting with a26b7c32d1be7c9d421bfdce725d558560ac9cec2f23be4fe8cf94e5d0870243 not found: ID does not exist" containerID="a26b7c32d1be7c9d421bfdce725d558560ac9cec2f23be4fe8cf94e5d0870243" Jul 23 00:21:38.791153 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:38.791107 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26b7c32d1be7c9d421bfdce725d558560ac9cec2f23be4fe8cf94e5d0870243"} err="failed to get container status \"a26b7c32d1be7c9d421bfdce725d558560ac9cec2f23be4fe8cf94e5d0870243\": rpc error: code = NotFound desc = could not find container \"a26b7c32d1be7c9d421bfdce725d558560ac9cec2f23be4fe8cf94e5d0870243\": container with ID starting with a26b7c32d1be7c9d421bfdce725d558560ac9cec2f23be4fe8cf94e5d0870243 not found: ID does not exist" Jul 23 00:21:38.797781 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:38.797761 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69c97bc454-4g54d"] Jul 23 00:21:38.801235 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:38.801216 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-69c97bc454-4g54d"] Jul 23 00:21:40.502093 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:40.502057 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cf33b38-60f5-4625-8503-8ee8561f4ff8" path="/var/lib/kubelet/pods/3cf33b38-60f5-4625-8503-8ee8561f4ff8/volumes" Jul 23 00:21:59.597107 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:59.597073 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:59.650878 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:59.650842 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:59.779881 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:59.779852 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:21:59.876961 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:21:59.876877 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:00.388271 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.388240 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:00.546348 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.546313 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d"] Jul 23 00:22:00.546667 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.546654 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cf33b38-60f5-4625-8503-8ee8561f4ff8" containerName="manager" Jul 23 00:22:00.546734 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.546668 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cf33b38-60f5-4625-8503-8ee8561f4ff8" containerName="manager" Jul 23 00:22:00.546791 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.546738 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cf33b38-60f5-4625-8503-8ee8561f4ff8" containerName="manager" Jul 23 00:22:00.554448 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.554427 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d" Jul 23 00:22:00.566429 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.564690 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 00:22:00.566429 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.564734 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 00:22:00.566429 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.564998 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 23 00:22:00.568969 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.568488 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d"] Jul 23 00:22:00.658407 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.658295 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/717116c9-0961-46e5-bff6-125cb9a7d4db-maas-api-tls\") pod \"maas-api-b877fd55f-hzg9d\" (UID: \"717116c9-0961-46e5-bff6-125cb9a7d4db\") " pod="odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d" Jul 23 00:22:00.658407 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.658358 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/717116c9-0961-46e5-bff6-125cb9a7d4db-openshift-service-ca\") pod \"maas-api-b877fd55f-hzg9d\" (UID: \"717116c9-0961-46e5-bff6-125cb9a7d4db\") " pod="odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d" Jul 23 00:22:00.658796 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.658478 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jxmz\" (UniqueName: \"kubernetes.io/projected/717116c9-0961-46e5-bff6-125cb9a7d4db-kube-api-access-4jxmz\") pod \"maas-api-b877fd55f-hzg9d\" (UID: \"717116c9-0961-46e5-bff6-125cb9a7d4db\") " pod="odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d" Jul 23 00:22:00.696364 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.696334 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:00.759367 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.759337 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/717116c9-0961-46e5-bff6-125cb9a7d4db-maas-api-tls\") pod \"maas-api-b877fd55f-hzg9d\" (UID: \"717116c9-0961-46e5-bff6-125cb9a7d4db\") " pod="odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d" Jul 23 00:22:00.759561 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.759378 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/717116c9-0961-46e5-bff6-125cb9a7d4db-openshift-service-ca\") pod \"maas-api-b877fd55f-hzg9d\" (UID: \"717116c9-0961-46e5-bff6-125cb9a7d4db\") " pod="odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d" Jul 23 00:22:00.759561 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.759438 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jxmz\" (UniqueName: \"kubernetes.io/projected/717116c9-0961-46e5-bff6-125cb9a7d4db-kube-api-access-4jxmz\") pod \"maas-api-b877fd55f-hzg9d\" (UID: \"717116c9-0961-46e5-bff6-125cb9a7d4db\") " pod="odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d" Jul 23 00:22:00.760068 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.760037 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/717116c9-0961-46e5-bff6-125cb9a7d4db-openshift-service-ca\") pod \"maas-api-b877fd55f-hzg9d\" (UID: \"717116c9-0961-46e5-bff6-125cb9a7d4db\") " pod="odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d" Jul 23 00:22:00.761826 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.761795 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/717116c9-0961-46e5-bff6-125cb9a7d4db-maas-api-tls\") pod \"maas-api-b877fd55f-hzg9d\" (UID: \"717116c9-0961-46e5-bff6-125cb9a7d4db\") " pod="odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d" Jul 23 00:22:00.768208 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.768183 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jxmz\" (UniqueName: \"kubernetes.io/projected/717116c9-0961-46e5-bff6-125cb9a7d4db-kube-api-access-4jxmz\") pod \"maas-api-b877fd55f-hzg9d\" (UID: \"717116c9-0961-46e5-bff6-125cb9a7d4db\") " pod="odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d" Jul 23 00:22:00.875106 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.875070 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d" Jul 23 00:22:00.991325 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:00.991268 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:01.017905 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:01.017866 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d"] Jul 23 00:22:01.021076 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:22:01.021045 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717116c9_0961_46e5_bff6_125cb9a7d4db.slice/crio-09954d4d3a2abfd29bf8a2ac4ad9d2f07bdcfb1386cfe5a1850bf913c36f0bca WatchSource:0}: Error finding container 09954d4d3a2abfd29bf8a2ac4ad9d2f07bdcfb1386cfe5a1850bf913c36f0bca: Status 404 returned error can't find the container with id 09954d4d3a2abfd29bf8a2ac4ad9d2f07bdcfb1386cfe5a1850bf913c36f0bca Jul 23 00:22:01.095916 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:01.095870 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:01.889956 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:01.889919 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d" event={"ID":"717116c9-0961-46e5-bff6-125cb9a7d4db","Type":"ContainerStarted","Data":"09954d4d3a2abfd29bf8a2ac4ad9d2f07bdcfb1386cfe5a1850bf913c36f0bca"} Jul 23 00:22:02.896110 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:02.896078 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d" event={"ID":"717116c9-0961-46e5-bff6-125cb9a7d4db","Type":"ContainerStarted","Data":"0a9fceb3ff27157f2c381001db38b0dbeb4f3167ee9c3737574ddc736206f243"} Jul 23 00:22:02.896546 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:02.896155 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d" Jul 23 00:22:02.912901 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:02.912853 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d" podStartSLOduration=1.133278983 podStartE2EDuration="2.91283622s" totalImagesPullingTime="1.779557237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:22:02.91199026 +0000 UTC m=+842.872728879" watchObservedRunningTime="2026-07-23 00:22:02.91283622 +0000 UTC m=+842.873574840" Jul 23 00:22:08.906917 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:08.906887 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-b877fd55f-hzg9d" Jul 23 00:22:09.675354 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:09.675323 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:09.929886 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:09.929815 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:10.029492 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:10.029464 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:26.644529 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:26.644497 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:32.053875 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:32.053840 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:22:45.751001 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:22:45.750956 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:00.422120 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:00.422088 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:23:00.422953 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:00.422929 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:23:06.427628 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.427592 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk"] Jul 23 00:23:06.430916 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.430901 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.433219 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.433199 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 23 00:23:06.433506 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.433491 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 00:23:06.434172 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.434153 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jghlj\"" Jul 23 00:23:06.434275 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.434189 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 00:23:06.443815 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.443793 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk"] Jul 23 00:23:06.456470 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.456450 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:06.520546 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.520513 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65aa11a5-cb74-4e56-ac70-22333c42f4a9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.520546 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.520545 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65aa11a5-cb74-4e56-ac70-22333c42f4a9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.520765 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.520567 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65aa11a5-cb74-4e56-ac70-22333c42f4a9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.520765 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.520625 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65aa11a5-cb74-4e56-ac70-22333c42f4a9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.520765 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.520641 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65aa11a5-cb74-4e56-ac70-22333c42f4a9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.520765 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.520693 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96q58\" (UniqueName: \"kubernetes.io/projected/65aa11a5-cb74-4e56-ac70-22333c42f4a9-kube-api-access-96q58\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.621639 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.621602 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96q58\" (UniqueName: \"kubernetes.io/projected/65aa11a5-cb74-4e56-ac70-22333c42f4a9-kube-api-access-96q58\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.621639 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.621641 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65aa11a5-cb74-4e56-ac70-22333c42f4a9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.621891 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.621662 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65aa11a5-cb74-4e56-ac70-22333c42f4a9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.621891 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.621682 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65aa11a5-cb74-4e56-ac70-22333c42f4a9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.621891 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.621732 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65aa11a5-cb74-4e56-ac70-22333c42f4a9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.621891 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.621751 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65aa11a5-cb74-4e56-ac70-22333c42f4a9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.622140 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.622119 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65aa11a5-cb74-4e56-ac70-22333c42f4a9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.622205 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.622139 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65aa11a5-cb74-4e56-ac70-22333c42f4a9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.622265 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.622206 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65aa11a5-cb74-4e56-ac70-22333c42f4a9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.623893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.623868 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65aa11a5-cb74-4e56-ac70-22333c42f4a9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.624191 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.624176 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65aa11a5-cb74-4e56-ac70-22333c42f4a9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.632480 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.632457 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96q58\" (UniqueName: \"kubernetes.io/projected/65aa11a5-cb74-4e56-ac70-22333c42f4a9-kube-api-access-96q58\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk\" (UID: \"65aa11a5-cb74-4e56-ac70-22333c42f4a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.741831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.741728 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:06.872454 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:06.872420 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk"] Jul 23 00:23:06.876031 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:23:06.876005 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65aa11a5_cb74_4e56_ac70_22333c42f4a9.slice/crio-0cadcfb83a0d2d4f9647af78636975270a14e8cecbec57f34107666fb77b2cab WatchSource:0}: Error finding container 0cadcfb83a0d2d4f9647af78636975270a14e8cecbec57f34107666fb77b2cab: Status 404 returned error can't find the container with id 0cadcfb83a0d2d4f9647af78636975270a14e8cecbec57f34107666fb77b2cab Jul 23 00:23:07.201596 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:07.201555 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" event={"ID":"65aa11a5-cb74-4e56-ac70-22333c42f4a9","Type":"ContainerStarted","Data":"0cadcfb83a0d2d4f9647af78636975270a14e8cecbec57f34107666fb77b2cab"} Jul 23 00:23:08.208022 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:08.207987 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" event={"ID":"65aa11a5-cb74-4e56-ac70-22333c42f4a9","Type":"ContainerStarted","Data":"c6ea1b7cb7070e71954d212069fe016c518057ae72d94b892483d5653cd8b9ac"} Jul 23 00:23:08.249838 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:08.249774 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" podStartSLOduration=1.219405732 podStartE2EDuration="2.249752438s" totalImagesPullingTime="1.030346706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:08.247761603 +0000 UTC m=+908.208500224" watchObservedRunningTime="2026-07-23 00:23:08.249752438 +0000 UTC m=+908.210491060" Jul 23 00:23:13.644720 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.644679 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd"] Jul 23 00:23:13.648125 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.648104 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.650898 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.650880 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 23 00:23:13.662852 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.662829 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd"] Jul 23 00:23:13.706892 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.706861 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:13.783984 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.783941 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab76740d-0364-4782-8c64-b1ecacd62265-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.783984 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.783985 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgrcg\" (UniqueName: \"kubernetes.io/projected/ab76740d-0364-4782-8c64-b1ecacd62265-kube-api-access-mgrcg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.784230 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.784022 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab76740d-0364-4782-8c64-b1ecacd62265-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.784230 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.784068 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab76740d-0364-4782-8c64-b1ecacd62265-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.784230 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.784095 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab76740d-0364-4782-8c64-b1ecacd62265-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.784230 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.784148 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab76740d-0364-4782-8c64-b1ecacd62265-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.885117 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.885084 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab76740d-0364-4782-8c64-b1ecacd62265-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.885292 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.885140 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab76740d-0364-4782-8c64-b1ecacd62265-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.885292 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.885159 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgrcg\" (UniqueName: \"kubernetes.io/projected/ab76740d-0364-4782-8c64-b1ecacd62265-kube-api-access-mgrcg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.885292 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.885182 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab76740d-0364-4782-8c64-b1ecacd62265-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.885292 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.885225 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab76740d-0364-4782-8c64-b1ecacd62265-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.885292 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.885243 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab76740d-0364-4782-8c64-b1ecacd62265-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.885666 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.885639 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab76740d-0364-4782-8c64-b1ecacd62265-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.885761 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.885700 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab76740d-0364-4782-8c64-b1ecacd62265-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.885821 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.885757 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab76740d-0364-4782-8c64-b1ecacd62265-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.888062 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.888039 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab76740d-0364-4782-8c64-b1ecacd62265-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.888248 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.888230 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab76740d-0364-4782-8c64-b1ecacd62265-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.894036 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.894016 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgrcg\" (UniqueName: \"kubernetes.io/projected/ab76740d-0364-4782-8c64-b1ecacd62265-kube-api-access-mgrcg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dqmfd\" (UID: \"ab76740d-0364-4782-8c64-b1ecacd62265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:13.958264 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:13.958223 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:14.093316 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:14.093289 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd"] Jul 23 00:23:14.094859 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:23:14.094829 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab76740d_0364_4782_8c64_b1ecacd62265.slice/crio-3c26e89a8577a2b5861e9f361f59209ddd4ea893ff93243ef23dfa7c128cebaf WatchSource:0}: Error finding container 3c26e89a8577a2b5861e9f361f59209ddd4ea893ff93243ef23dfa7c128cebaf: Status 404 returned error can't find the container with id 3c26e89a8577a2b5861e9f361f59209ddd4ea893ff93243ef23dfa7c128cebaf Jul 23 00:23:14.236096 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:14.236026 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" event={"ID":"ab76740d-0364-4782-8c64-b1ecacd62265","Type":"ContainerStarted","Data":"3c26e89a8577a2b5861e9f361f59209ddd4ea893ff93243ef23dfa7c128cebaf"} Jul 23 00:23:15.241689 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:15.241655 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" event={"ID":"ab76740d-0364-4782-8c64-b1ecacd62265","Type":"ContainerStarted","Data":"06e035158bafa3c3f6b1bc41f3f71029711f315d52e4d418fea3d0b3556129c9"} Jul 23 00:23:15.258781 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:15.258737 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" podStartSLOduration=2.098584174 podStartE2EDuration="2.258723641s" totalImagesPullingTime="160.139467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:15.257031637 +0000 UTC m=+915.217770257" watchObservedRunningTime="2026-07-23 00:23:15.258723641 +0000 UTC m=+915.219462260" Jul 23 00:23:16.742833 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:16.742794 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:16.743234 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:16.742858 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:16.755215 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:16.755189 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:17.263955 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:17.263925 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk" Jul 23 00:23:23.958671 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:23.958633 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:23.959068 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:23.958690 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:23.971039 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:23.971019 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:24.294128 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:24.294053 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dqmfd" Jul 23 00:23:45.049879 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:45.049845 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:57.026659 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:57.026612 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:57.081340 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:23:57.081295 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:43.320333 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:43.320254 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-86d8db559f-zjzj8"] Jul 23 00:24:43.323751 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:43.323733 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86d8db559f-zjzj8" Jul 23 00:24:43.331154 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:43.330888 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86d8db559f-zjzj8"] Jul 23 00:24:43.396575 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:43.396539 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2a881541-ed40-44b0-8709-2e1af5c1204e-tls-cert\") pod \"authorino-86d8db559f-zjzj8\" (UID: \"2a881541-ed40-44b0-8709-2e1af5c1204e\") " pod="kuadrant-system/authorino-86d8db559f-zjzj8" Jul 23 00:24:43.396575 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:43.396580 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2xpc\" (UniqueName: \"kubernetes.io/projected/2a881541-ed40-44b0-8709-2e1af5c1204e-kube-api-access-k2xpc\") pod \"authorino-86d8db559f-zjzj8\" (UID: \"2a881541-ed40-44b0-8709-2e1af5c1204e\") " pod="kuadrant-system/authorino-86d8db559f-zjzj8" Jul 23 00:24:43.497112 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:43.497076 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2a881541-ed40-44b0-8709-2e1af5c1204e-tls-cert\") pod \"authorino-86d8db559f-zjzj8\" (UID: \"2a881541-ed40-44b0-8709-2e1af5c1204e\") " pod="kuadrant-system/authorino-86d8db559f-zjzj8" Jul 23 00:24:43.497112 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:43.497116 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xpc\" (UniqueName: \"kubernetes.io/projected/2a881541-ed40-44b0-8709-2e1af5c1204e-kube-api-access-k2xpc\") pod \"authorino-86d8db559f-zjzj8\" (UID: \"2a881541-ed40-44b0-8709-2e1af5c1204e\") " pod="kuadrant-system/authorino-86d8db559f-zjzj8" Jul 23 00:24:43.499703 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:43.499670 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2a881541-ed40-44b0-8709-2e1af5c1204e-tls-cert\") pod \"authorino-86d8db559f-zjzj8\" (UID: \"2a881541-ed40-44b0-8709-2e1af5c1204e\") " pod="kuadrant-system/authorino-86d8db559f-zjzj8" Jul 23 00:24:43.504811 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:43.504794 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2xpc\" (UniqueName: \"kubernetes.io/projected/2a881541-ed40-44b0-8709-2e1af5c1204e-kube-api-access-k2xpc\") pod \"authorino-86d8db559f-zjzj8\" (UID: \"2a881541-ed40-44b0-8709-2e1af5c1204e\") " pod="kuadrant-system/authorino-86d8db559f-zjzj8" Jul 23 00:24:43.633723 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:43.633646 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86d8db559f-zjzj8" Jul 23 00:24:43.764336 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:43.764312 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86d8db559f-zjzj8"] Jul 23 00:24:43.765999 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:24:43.765971 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a881541_ed40_44b0_8709_2e1af5c1204e.slice/crio-1c2d8e03cd5e8ca1afb9565872d95530da6056d472da514fa61128ac946c49c3 WatchSource:0}: Error finding container 1c2d8e03cd5e8ca1afb9565872d95530da6056d472da514fa61128ac946c49c3: Status 404 returned error can't find the container with id 1c2d8e03cd5e8ca1afb9565872d95530da6056d472da514fa61128ac946c49c3 Jul 23 00:24:44.661880 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:44.661846 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86d8db559f-zjzj8" event={"ID":"2a881541-ed40-44b0-8709-2e1af5c1204e","Type":"ContainerStarted","Data":"fc9b00c06e6ecd94bd350f183715f2a88d48c43c7e44fb33313a99226e9fd58c"} Jul 23 00:24:44.661880 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:44.661886 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86d8db559f-zjzj8" event={"ID":"2a881541-ed40-44b0-8709-2e1af5c1204e","Type":"ContainerStarted","Data":"1c2d8e03cd5e8ca1afb9565872d95530da6056d472da514fa61128ac946c49c3"} Jul 23 00:24:44.681670 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:44.681622 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-86d8db559f-zjzj8" podStartSLOduration=1.293417168 podStartE2EDuration="1.681605089s" totalImagesPullingTime="388.187921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:24:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:24:44.680584439 +0000 UTC m=+1004.641323059" watchObservedRunningTime="2026-07-23 00:24:44.681605089 +0000 UTC m=+1004.642343709" Jul 23 00:24:44.713653 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:44.713619 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7558cfdcd-bnm8d"] Jul 23 00:24:44.714065 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:44.714031 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7558cfdcd-bnm8d" podUID="e6fbb509-6676-44ad-8809-432f8566fce7" containerName="authorino" containerID="cri-o://f5a7c1bd6c85324ce0bb7fbbcd079c7c4c3c768c98f0d76b92516b8908aae71c" gracePeriod=30 Jul 23 00:24:44.991524 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:44.991497 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7558cfdcd-bnm8d" Jul 23 00:24:45.015332 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:45.015301 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e6fbb509-6676-44ad-8809-432f8566fce7-tls-cert\") pod \"e6fbb509-6676-44ad-8809-432f8566fce7\" (UID: \"e6fbb509-6676-44ad-8809-432f8566fce7\") " Jul 23 00:24:45.015553 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:45.015372 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58knp\" (UniqueName: \"kubernetes.io/projected/e6fbb509-6676-44ad-8809-432f8566fce7-kube-api-access-58knp\") pod \"e6fbb509-6676-44ad-8809-432f8566fce7\" (UID: \"e6fbb509-6676-44ad-8809-432f8566fce7\") " Jul 23 00:24:45.018136 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:45.018104 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fbb509-6676-44ad-8809-432f8566fce7-kube-api-access-58knp" (OuterVolumeSpecName: "kube-api-access-58knp") pod "e6fbb509-6676-44ad-8809-432f8566fce7" (UID: "e6fbb509-6676-44ad-8809-432f8566fce7"). InnerVolumeSpecName "kube-api-access-58knp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:24:45.026355 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:45.026325 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6fbb509-6676-44ad-8809-432f8566fce7-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e6fbb509-6676-44ad-8809-432f8566fce7" (UID: "e6fbb509-6676-44ad-8809-432f8566fce7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:24:45.117008 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:45.116975 2555 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e6fbb509-6676-44ad-8809-432f8566fce7-tls-cert\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:24:45.117008 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:45.117003 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58knp\" (UniqueName: \"kubernetes.io/projected/e6fbb509-6676-44ad-8809-432f8566fce7-kube-api-access-58knp\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:24:45.259470 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:45.259339 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:45.349004 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:45.348970 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:45.669351 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:45.669322 2555 generic.go:356] "Generic (PLEG): container finished" podID="e6fbb509-6676-44ad-8809-432f8566fce7" containerID="f5a7c1bd6c85324ce0bb7fbbcd079c7c4c3c768c98f0d76b92516b8908aae71c" exitCode=0 Jul 23 00:24:45.669783 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:45.669384 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7558cfdcd-bnm8d" Jul 23 00:24:45.669783 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:45.669401 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7558cfdcd-bnm8d" event={"ID":"e6fbb509-6676-44ad-8809-432f8566fce7","Type":"ContainerDied","Data":"f5a7c1bd6c85324ce0bb7fbbcd079c7c4c3c768c98f0d76b92516b8908aae71c"} Jul 23 00:24:45.669783 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:45.669469 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7558cfdcd-bnm8d" event={"ID":"e6fbb509-6676-44ad-8809-432f8566fce7","Type":"ContainerDied","Data":"7938168bc496fe8b75529204328f5feb814b1c125a9d072dab2229fedff030e0"} Jul 23 00:24:45.669783 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:45.669498 2555 scope.go:117] "RemoveContainer" containerID="f5a7c1bd6c85324ce0bb7fbbcd079c7c4c3c768c98f0d76b92516b8908aae71c" Jul 23 00:24:45.679148 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:45.679127 2555 scope.go:117] "RemoveContainer" containerID="f5a7c1bd6c85324ce0bb7fbbcd079c7c4c3c768c98f0d76b92516b8908aae71c" Jul 23 00:24:45.679520 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:24:45.679495 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a7c1bd6c85324ce0bb7fbbcd079c7c4c3c768c98f0d76b92516b8908aae71c\": container with ID starting with f5a7c1bd6c85324ce0bb7fbbcd079c7c4c3c768c98f0d76b92516b8908aae71c not found: ID does not exist" containerID="f5a7c1bd6c85324ce0bb7fbbcd079c7c4c3c768c98f0d76b92516b8908aae71c" Jul 23 00:24:45.679610 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:45.679535 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a7c1bd6c85324ce0bb7fbbcd079c7c4c3c768c98f0d76b92516b8908aae71c"} err="failed to get container status \"f5a7c1bd6c85324ce0bb7fbbcd079c7c4c3c768c98f0d76b92516b8908aae71c\": rpc error: code = NotFound desc = could not find container \"f5a7c1bd6c85324ce0bb7fbbcd079c7c4c3c768c98f0d76b92516b8908aae71c\": container with ID starting with f5a7c1bd6c85324ce0bb7fbbcd079c7c4c3c768c98f0d76b92516b8908aae71c not found: ID does not exist" Jul 23 00:24:45.693785 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:45.693759 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7558cfdcd-bnm8d"] Jul 23 00:24:45.703083 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:45.703055 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7558cfdcd-bnm8d"] Jul 23 00:24:46.489588 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:46.489555 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:46.503887 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:46.503854 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6fbb509-6676-44ad-8809-432f8566fce7" path="/var/lib/kubelet/pods/e6fbb509-6676-44ad-8809-432f8566fce7/volumes" Jul 23 00:24:46.577804 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:24:46.577764 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:25:56.812059 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:25:56.811931 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:33.625766 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:26:33.625729 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:50.884687 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:26:50.884654 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:52.130789 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:26:52.130755 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:27.904843 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:27:27.904745 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:27.999828 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:27:27.999793 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:28.140004 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:27:28.139976 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:34.067581 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:27:34.067548 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:34.176193 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:27:34.176161 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:34.269919 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:27:34.269891 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:34.466503 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:27:34.466470 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:40.056654 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:27:40.056623 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:40.157670 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:27:40.157635 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:42.184406 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:27:42.184367 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:00.450128 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:28:00.450094 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:28:00.452684 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:28:00.452657 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:32:51.546667 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:32:51.546638 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:52.718791 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:32:52.718760 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:00.477202 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:33:00.477171 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:33:00.487424 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:33:00.487381 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:34:18.811711 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:34:18.811679 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:37.244010 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:34:37.243973 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:32.569710 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:32.569671 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:42.522219 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.522188 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr"] Jul 23 00:35:42.522759 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.522740 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6fbb509-6676-44ad-8809-432f8566fce7" containerName="authorino" Jul 23 00:35:42.522843 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.522762 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6fbb509-6676-44ad-8809-432f8566fce7" containerName="authorino" Jul 23 00:35:42.522904 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.522881 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6fbb509-6676-44ad-8809-432f8566fce7" containerName="authorino" Jul 23 00:35:42.526164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.526144 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.528710 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.528690 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c35c73c7-kserve-self-signed-certs\"" Jul 23 00:35:42.539521 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.539494 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr"] Jul 23 00:35:42.565962 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.565931 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:42.614530 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.614493 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-dshm\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.614718 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.614561 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-home\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.614718 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.614585 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfbj7\" (UniqueName: \"kubernetes.io/projected/7a367f65-617f-4d05-99c0-a548532c8ce6-kube-api-access-hfbj7\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.614718 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.614706 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-model-cache\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.614875 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.614755 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-tmp-dir\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.614917 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.614874 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a367f65-617f-4d05-99c0-a548532c8ce6-tls-certs\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.715622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.715584 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-home\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.715622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.715624 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfbj7\" (UniqueName: \"kubernetes.io/projected/7a367f65-617f-4d05-99c0-a548532c8ce6-kube-api-access-hfbj7\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.715856 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.715644 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-model-cache\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.715856 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.715674 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-tmp-dir\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.715856 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.715747 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a367f65-617f-4d05-99c0-a548532c8ce6-tls-certs\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.715856 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.715771 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-dshm\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.716070 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.715963 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-home\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.716070 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.716029 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-model-cache\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.716183 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.716117 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-tmp-dir\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.718065 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.718041 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-dshm\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.718183 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.718132 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a367f65-617f-4d05-99c0-a548532c8ce6-tls-certs\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.723976 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.723954 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfbj7\" (UniqueName: \"kubernetes.io/projected/7a367f65-617f-4d05-99c0-a548532c8ce6-kube-api-access-hfbj7\") pod \"e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.836194 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.836103 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:42.973891 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:42.973691 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr"] Jul 23 00:35:42.976216 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:35:42.976187 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a367f65_617f_4d05_99c0_a548532c8ce6.slice/crio-5998391f5204c5b1f771412ef853f0511644db19f3b56fbdfab27de849cd808e WatchSource:0}: Error finding container 5998391f5204c5b1f771412ef853f0511644db19f3b56fbdfab27de849cd808e: Status 404 returned error can't find the container with id 5998391f5204c5b1f771412ef853f0511644db19f3b56fbdfab27de849cd808e Jul 23 00:35:43.717792 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:43.717760 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" event={"ID":"7a367f65-617f-4d05-99c0-a548532c8ce6","Type":"ContainerStarted","Data":"8ce735b0b9db49099f44f2e6f40e5faa08ffb36b929520e53e63dbf25c5c202f"} Jul 23 00:35:43.717792 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:43.717797 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" event={"ID":"7a367f65-617f-4d05-99c0-a548532c8ce6","Type":"ContainerStarted","Data":"5998391f5204c5b1f771412ef853f0511644db19f3b56fbdfab27de849cd808e"} Jul 23 00:35:43.734883 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:43.734833 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" podStartSLOduration=1.73481811 podStartE2EDuration="1.73481811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:35:43.733128193 +0000 UTC m=+1663.693866807" watchObservedRunningTime="2026-07-23 00:35:43.73481811 +0000 UTC m=+1663.695556731" Jul 23 00:35:52.836815 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:52.836775 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:52.837230 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:52.836833 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:52.849260 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:52.849235 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:35:53.778938 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:35:53.778902 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:36:23.530940 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:23.530818 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:25.014219 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:25.014185 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr"] Jul 23 00:36:25.035963 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:25.035910 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:40.035578 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.035535 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" podUID="7a367f65-617f-4d05-99c0-a548532c8ce6" containerName="main" containerID="cri-o://8ce735b0b9db49099f44f2e6f40e5faa08ffb36b929520e53e63dbf25c5c202f" gracePeriod=45 Jul 23 00:36:40.284509 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.284485 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:36:40.431075 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.431039 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a367f65-617f-4d05-99c0-a548532c8ce6-tls-certs\") pod \"7a367f65-617f-4d05-99c0-a548532c8ce6\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " Jul 23 00:36:40.431237 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.431089 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfbj7\" (UniqueName: \"kubernetes.io/projected/7a367f65-617f-4d05-99c0-a548532c8ce6-kube-api-access-hfbj7\") pod \"7a367f65-617f-4d05-99c0-a548532c8ce6\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " Jul 23 00:36:40.431237 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.431119 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-dshm\") pod \"7a367f65-617f-4d05-99c0-a548532c8ce6\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " Jul 23 00:36:40.431237 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.431151 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-model-cache\") pod \"7a367f65-617f-4d05-99c0-a548532c8ce6\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " Jul 23 00:36:40.431237 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.431177 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-home\") pod \"7a367f65-617f-4d05-99c0-a548532c8ce6\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " Jul 23 00:36:40.431413 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.431254 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-tmp-dir\") pod \"7a367f65-617f-4d05-99c0-a548532c8ce6\" (UID: \"7a367f65-617f-4d05-99c0-a548532c8ce6\") " Jul 23 00:36:40.431475 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.431434 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a367f65-617f-4d05-99c0-a548532c8ce6" (UID: "7a367f65-617f-4d05-99c0-a548532c8ce6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:36:40.431537 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.431484 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-home" (OuterVolumeSpecName: "home") pod "7a367f65-617f-4d05-99c0-a548532c8ce6" (UID: "7a367f65-617f-4d05-99c0-a548532c8ce6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:36:40.431625 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.431591 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a367f65-617f-4d05-99c0-a548532c8ce6" (UID: "7a367f65-617f-4d05-99c0-a548532c8ce6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:36:40.431625 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.431615 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-model-cache\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:36:40.431712 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.431633 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-home\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:36:40.433295 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.433276 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a367f65-617f-4d05-99c0-a548532c8ce6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a367f65-617f-4d05-99c0-a548532c8ce6" (UID: "7a367f65-617f-4d05-99c0-a548532c8ce6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:36:40.433581 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.433568 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-dshm" (OuterVolumeSpecName: "dshm") pod "7a367f65-617f-4d05-99c0-a548532c8ce6" (UID: "7a367f65-617f-4d05-99c0-a548532c8ce6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:36:40.433635 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.433600 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a367f65-617f-4d05-99c0-a548532c8ce6-kube-api-access-hfbj7" (OuterVolumeSpecName: "kube-api-access-hfbj7") pod "7a367f65-617f-4d05-99c0-a548532c8ce6" (UID: "7a367f65-617f-4d05-99c0-a548532c8ce6"). InnerVolumeSpecName "kube-api-access-hfbj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:36:40.532242 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.532215 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfbj7\" (UniqueName: \"kubernetes.io/projected/7a367f65-617f-4d05-99c0-a548532c8ce6-kube-api-access-hfbj7\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:36:40.532242 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.532241 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-dshm\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:36:40.532440 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.532276 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a367f65-617f-4d05-99c0-a548532c8ce6-tmp-dir\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:36:40.532440 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.532288 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a367f65-617f-4d05-99c0-a548532c8ce6-tls-certs\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:36:40.986149 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.986110 2555 generic.go:356] "Generic (PLEG): container finished" podID="7a367f65-617f-4d05-99c0-a548532c8ce6" containerID="8ce735b0b9db49099f44f2e6f40e5faa08ffb36b929520e53e63dbf25c5c202f" exitCode=0 Jul 23 00:36:40.986327 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.986173 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" Jul 23 00:36:40.986327 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.986168 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" event={"ID":"7a367f65-617f-4d05-99c0-a548532c8ce6","Type":"ContainerDied","Data":"8ce735b0b9db49099f44f2e6f40e5faa08ffb36b929520e53e63dbf25c5c202f"} Jul 23 00:36:40.986327 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.986281 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr" event={"ID":"7a367f65-617f-4d05-99c0-a548532c8ce6","Type":"ContainerDied","Data":"5998391f5204c5b1f771412ef853f0511644db19f3b56fbdfab27de849cd808e"} Jul 23 00:36:40.986327 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.986310 2555 scope.go:117] "RemoveContainer" containerID="8ce735b0b9db49099f44f2e6f40e5faa08ffb36b929520e53e63dbf25c5c202f" Jul 23 00:36:40.995081 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.995059 2555 scope.go:117] "RemoveContainer" containerID="8ce735b0b9db49099f44f2e6f40e5faa08ffb36b929520e53e63dbf25c5c202f" Jul 23 00:36:40.995336 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:36:40.995318 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce735b0b9db49099f44f2e6f40e5faa08ffb36b929520e53e63dbf25c5c202f\": container with ID starting with 8ce735b0b9db49099f44f2e6f40e5faa08ffb36b929520e53e63dbf25c5c202f not found: ID does not exist" containerID="8ce735b0b9db49099f44f2e6f40e5faa08ffb36b929520e53e63dbf25c5c202f" Jul 23 00:36:40.995379 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:40.995346 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce735b0b9db49099f44f2e6f40e5faa08ffb36b929520e53e63dbf25c5c202f"} err="failed to get container status \"8ce735b0b9db49099f44f2e6f40e5faa08ffb36b929520e53e63dbf25c5c202f\": rpc error: code = NotFound desc = could not find container \"8ce735b0b9db49099f44f2e6f40e5faa08ffb36b929520e53e63dbf25c5c202f\": container with ID starting with 8ce735b0b9db49099f44f2e6f40e5faa08ffb36b929520e53e63dbf25c5c202f not found: ID does not exist" Jul 23 00:36:41.002507 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:41.002486 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr"] Jul 23 00:36:41.006736 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:41.006714 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c35c73c7-kserve-748847dbff-6z7wr"] Jul 23 00:36:42.502243 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:36:42.502209 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a367f65-617f-4d05-99c0-a548532c8ce6" path="/var/lib/kubelet/pods/7a367f65-617f-4d05-99c0-a548532c8ce6/volumes" Jul 23 00:38:00.509240 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:38:00.509204 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:38:00.516047 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:38:00.516018 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:38:46.710374 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:38:46.710333 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:38:48.347117 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:38:48.347079 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:39:07.655668 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:39:07.655601 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:39:39.095832 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:39:39.095754 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:40:02.774878 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:02.774842 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:40:05.846922 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:05.846888 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:40:05.949071 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:05.949043 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:40:06.323128 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:06.323093 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:40:12.055985 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:12.055948 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:40:12.156850 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:12.156814 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:40:12.310281 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:12.310198 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:40:12.626847 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:12.626750 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:40:13.060723 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:13.060686 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:40:14.891617 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:14.891578 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:40:14.968595 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:14.968563 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:40:14.972718 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:14.972689 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw"] Jul 23 00:40:14.973241 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:14.973220 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a367f65-617f-4d05-99c0-a548532c8ce6" containerName="main" Jul 23 00:40:14.973241 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:14.973241 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a367f65-617f-4d05-99c0-a548532c8ce6" containerName="main" Jul 23 00:40:14.973447 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:14.973339 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a367f65-617f-4d05-99c0-a548532c8ce6" containerName="main" Jul 23 00:40:14.976868 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:14.976847 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" Jul 23 00:40:14.980092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:14.980070 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3eadb28c-serving-cert\"" Jul 23 00:40:14.980347 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:14.980326 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gktdb\"" Jul 23 00:40:14.987903 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:14.987879 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw"] Jul 23 00:40:15.038134 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:15.038089 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:40:15.047002 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:15.046975 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e125a47-173d-47f4-849f-f79a2c517417-openshift-service-ca\") pod \"maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw\" (UID: \"8e125a47-173d-47f4-849f-f79a2c517417\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" Jul 23 00:40:15.047146 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:15.047011 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e125a47-173d-47f4-849f-f79a2c517417-maas-api-tls\") pod \"maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw\" (UID: \"8e125a47-173d-47f4-849f-f79a2c517417\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" Jul 23 00:40:15.047146 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:15.047047 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-844wx\" (UniqueName: \"kubernetes.io/projected/8e125a47-173d-47f4-849f-f79a2c517417-kube-api-access-844wx\") pod \"maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw\" (UID: \"8e125a47-173d-47f4-849f-f79a2c517417\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" Jul 23 00:40:15.148504 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:15.148380 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e125a47-173d-47f4-849f-f79a2c517417-openshift-service-ca\") pod \"maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw\" (UID: \"8e125a47-173d-47f4-849f-f79a2c517417\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" Jul 23 00:40:15.148504 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:15.148470 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e125a47-173d-47f4-849f-f79a2c517417-maas-api-tls\") pod \"maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw\" (UID: \"8e125a47-173d-47f4-849f-f79a2c517417\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" Jul 23 00:40:15.148747 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:15.148528 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-844wx\" (UniqueName: \"kubernetes.io/projected/8e125a47-173d-47f4-849f-f79a2c517417-kube-api-access-844wx\") pod \"maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw\" (UID: \"8e125a47-173d-47f4-849f-f79a2c517417\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" Jul 23 00:40:15.148747 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:40:15.148588 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-serving-cert: secret "maas-api-e2e-ait-3eadb28c-serving-cert" not found Jul 23 00:40:15.148747 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:40:15.148662 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e125a47-173d-47f4-849f-f79a2c517417-maas-api-tls podName:8e125a47-173d-47f4-849f-f79a2c517417 nodeName:}" failed. No retries permitted until 2026-07-23 00:40:15.648640505 +0000 UTC m=+1935.609379104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e125a47-173d-47f4-849f-f79a2c517417-maas-api-tls") pod "maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" (UID: "8e125a47-173d-47f4-849f-f79a2c517417") : secret "maas-api-e2e-ait-3eadb28c-serving-cert" not found Jul 23 00:40:15.149103 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:15.149085 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e125a47-173d-47f4-849f-f79a2c517417-openshift-service-ca\") pod \"maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw\" (UID: \"8e125a47-173d-47f4-849f-f79a2c517417\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" Jul 23 00:40:15.157565 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:15.157542 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-844wx\" (UniqueName: \"kubernetes.io/projected/8e125a47-173d-47f4-849f-f79a2c517417-kube-api-access-844wx\") pod \"maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw\" (UID: \"8e125a47-173d-47f4-849f-f79a2c517417\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" Jul 23 00:40:15.534151 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:15.534120 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:40:15.653370 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:15.653338 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e125a47-173d-47f4-849f-f79a2c517417-maas-api-tls\") pod \"maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw\" (UID: \"8e125a47-173d-47f4-849f-f79a2c517417\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" Jul 23 00:40:15.655729 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:15.655707 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e125a47-173d-47f4-849f-f79a2c517417-maas-api-tls\") pod \"maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw\" (UID: \"8e125a47-173d-47f4-849f-f79a2c517417\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" Jul 23 00:40:15.669520 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:15.669493 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:40:15.770442 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:15.770411 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:40:15.888170 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:15.888075 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" Jul 23 00:40:16.022343 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:16.022320 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw"] Jul 23 00:40:16.024537 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:40:16.024509 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e125a47_173d_47f4_849f_f79a2c517417.slice/crio-1ecf934032f6d679155c3be4ff69444a3fb909f5e65f8f86a015855f853e1961 WatchSource:0}: Error finding container 1ecf934032f6d679155c3be4ff69444a3fb909f5e65f8f86a015855f853e1961: Status 404 returned error can't find the container with id 1ecf934032f6d679155c3be4ff69444a3fb909f5e65f8f86a015855f853e1961 Jul 23 00:40:16.025865 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:16.025845 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:40:17.032632 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:17.032594 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" event={"ID":"8e125a47-173d-47f4-849f-f79a2c517417","Type":"ContainerStarted","Data":"9a67e53f8580dbc6a56e005e08844a9abfbb58934274f7a5c041146da664d4bc"} Jul 23 00:40:17.033064 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:17.032652 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" event={"ID":"8e125a47-173d-47f4-849f-f79a2c517417","Type":"ContainerStarted","Data":"1ecf934032f6d679155c3be4ff69444a3fb909f5e65f8f86a015855f853e1961"} Jul 23 00:40:17.033064 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:17.032755 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" Jul 23 00:40:17.049237 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:17.049190 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" podStartSLOduration=2.635765778 podStartE2EDuration="3.049173266s" totalImagesPullingTime="413.407488ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:40:17.047590183 +0000 UTC m=+1937.008328827" watchObservedRunningTime="2026-07-23 00:40:17.049173266 +0000 UTC m=+1937.009911884" Jul 23 00:40:23.044851 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:40:23.044820 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" Jul 23 00:41:00.892744 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:00.892664 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw"] Jul 23 00:41:00.893260 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:00.892875 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" podUID="8e125a47-173d-47f4-849f-f79a2c517417" containerName="maas-api" containerID="cri-o://9a67e53f8580dbc6a56e005e08844a9abfbb58934274f7a5c041146da664d4bc" gracePeriod=30 Jul 23 00:41:00.914501 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:00.914474 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:01.141280 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.141257 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" Jul 23 00:41:01.251315 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.251283 2555 generic.go:356] "Generic (PLEG): container finished" podID="8e125a47-173d-47f4-849f-f79a2c517417" containerID="9a67e53f8580dbc6a56e005e08844a9abfbb58934274f7a5c041146da664d4bc" exitCode=0 Jul 23 00:41:01.251529 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.251322 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" event={"ID":"8e125a47-173d-47f4-849f-f79a2c517417","Type":"ContainerDied","Data":"9a67e53f8580dbc6a56e005e08844a9abfbb58934274f7a5c041146da664d4bc"} Jul 23 00:41:01.251529 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.251334 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" Jul 23 00:41:01.251529 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.251364 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw" event={"ID":"8e125a47-173d-47f4-849f-f79a2c517417","Type":"ContainerDied","Data":"1ecf934032f6d679155c3be4ff69444a3fb909f5e65f8f86a015855f853e1961"} Jul 23 00:41:01.251529 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.251405 2555 scope.go:117] "RemoveContainer" containerID="9a67e53f8580dbc6a56e005e08844a9abfbb58934274f7a5c041146da664d4bc" Jul 23 00:41:01.256431 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.256376 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e125a47-173d-47f4-849f-f79a2c517417-maas-api-tls\") pod \"8e125a47-173d-47f4-849f-f79a2c517417\" (UID: \"8e125a47-173d-47f4-849f-f79a2c517417\") " Jul 23 00:41:01.256559 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.256461 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e125a47-173d-47f4-849f-f79a2c517417-openshift-service-ca\") pod \"8e125a47-173d-47f4-849f-f79a2c517417\" (UID: \"8e125a47-173d-47f4-849f-f79a2c517417\") " Jul 23 00:41:01.256559 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.256482 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-844wx\" (UniqueName: \"kubernetes.io/projected/8e125a47-173d-47f4-849f-f79a2c517417-kube-api-access-844wx\") pod \"8e125a47-173d-47f4-849f-f79a2c517417\" (UID: \"8e125a47-173d-47f4-849f-f79a2c517417\") " Jul 23 00:41:01.256831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.256803 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e125a47-173d-47f4-849f-f79a2c517417-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e125a47-173d-47f4-849f-f79a2c517417" (UID: "8e125a47-173d-47f4-849f-f79a2c517417"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:41:01.258617 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.258589 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e125a47-173d-47f4-849f-f79a2c517417-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e125a47-173d-47f4-849f-f79a2c517417" (UID: "8e125a47-173d-47f4-849f-f79a2c517417"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:41:01.258725 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.258706 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e125a47-173d-47f4-849f-f79a2c517417-kube-api-access-844wx" (OuterVolumeSpecName: "kube-api-access-844wx") pod "8e125a47-173d-47f4-849f-f79a2c517417" (UID: "8e125a47-173d-47f4-849f-f79a2c517417"). InnerVolumeSpecName "kube-api-access-844wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:41:01.260177 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.260156 2555 scope.go:117] "RemoveContainer" containerID="9a67e53f8580dbc6a56e005e08844a9abfbb58934274f7a5c041146da664d4bc" Jul 23 00:41:01.260491 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:41:01.260473 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a67e53f8580dbc6a56e005e08844a9abfbb58934274f7a5c041146da664d4bc\": container with ID starting with 9a67e53f8580dbc6a56e005e08844a9abfbb58934274f7a5c041146da664d4bc not found: ID does not exist" containerID="9a67e53f8580dbc6a56e005e08844a9abfbb58934274f7a5c041146da664d4bc" Jul 23 00:41:01.260571 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.260498 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a67e53f8580dbc6a56e005e08844a9abfbb58934274f7a5c041146da664d4bc"} err="failed to get container status \"9a67e53f8580dbc6a56e005e08844a9abfbb58934274f7a5c041146da664d4bc\": rpc error: code = NotFound desc = could not find container \"9a67e53f8580dbc6a56e005e08844a9abfbb58934274f7a5c041146da664d4bc\": container with ID starting with 9a67e53f8580dbc6a56e005e08844a9abfbb58934274f7a5c041146da664d4bc not found: ID does not exist" Jul 23 00:41:01.357738 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.357705 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e125a47-173d-47f4-849f-f79a2c517417-openshift-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:41:01.357738 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.357734 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-844wx\" (UniqueName: \"kubernetes.io/projected/8e125a47-173d-47f4-849f-f79a2c517417-kube-api-access-844wx\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:41:01.357738 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.357743 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e125a47-173d-47f4-849f-f79a2c517417-maas-api-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:41:01.577687 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.577650 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw"] Jul 23 00:41:01.580438 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:01.580383 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eadb28c-7bb9b6dccb-l8vpw"] Jul 23 00:41:02.503431 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:02.503374 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e125a47-173d-47f4-849f-f79a2c517417" path="/var/lib/kubelet/pods/8e125a47-173d-47f4-849f-f79a2c517417/volumes" Jul 23 00:41:05.596864 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:05.596830 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:05.828316 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:05.828286 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:06.012861 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:06.012832 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:06.295306 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:06.295217 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:06.392930 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:06.392895 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:06.491106 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:06.491075 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:06.808830 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:06.808797 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:07.121431 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:07.121328 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:08.869558 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:08.869522 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:09.050914 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.050878 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g"] Jul 23 00:41:09.051445 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.051429 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e125a47-173d-47f4-849f-f79a2c517417" containerName="maas-api" Jul 23 00:41:09.051508 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.051448 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e125a47-173d-47f4-849f-f79a2c517417" containerName="maas-api" Jul 23 00:41:09.051553 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.051542 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e125a47-173d-47f4-849f-f79a2c517417" containerName="maas-api" Jul 23 00:41:09.054954 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.054933 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" Jul 23 00:41:09.057464 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.057442 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-72954db1-serving-cert\"" Jul 23 00:41:09.057583 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.057445 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gktdb\"" Jul 23 00:41:09.063354 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.063326 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g"] Jul 23 00:41:09.084437 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.084407 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:09.132327 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.132232 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5mh2\" (UniqueName: \"kubernetes.io/projected/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-kube-api-access-s5mh2\") pod \"maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g\" (UID: \"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" Jul 23 00:41:09.132327 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.132303 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-openshift-service-ca\") pod \"maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g\" (UID: \"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" Jul 23 00:41:09.132550 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.132359 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-maas-api-tls\") pod \"maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g\" (UID: \"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" Jul 23 00:41:09.233604 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.233569 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5mh2\" (UniqueName: \"kubernetes.io/projected/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-kube-api-access-s5mh2\") pod \"maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g\" (UID: \"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" Jul 23 00:41:09.233863 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.233730 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-openshift-service-ca\") pod \"maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g\" (UID: \"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" Jul 23 00:41:09.233863 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.233786 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-maas-api-tls\") pod \"maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g\" (UID: \"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" Jul 23 00:41:09.234011 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:41:09.233918 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-serving-cert: secret "maas-api-e2e-migrate-72954db1-serving-cert" not found Jul 23 00:41:09.234011 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:41:09.233979 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-maas-api-tls podName:cbc8664d-7582-4ffa-bcc8-7c470bd6d17e nodeName:}" failed. No retries permitted until 2026-07-23 00:41:09.733964475 +0000 UTC m=+1989.694703075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-maas-api-tls") pod "maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" (UID: "cbc8664d-7582-4ffa-bcc8-7c470bd6d17e") : secret "maas-api-e2e-migrate-72954db1-serving-cert" not found Jul 23 00:41:09.234333 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.234316 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-openshift-service-ca\") pod \"maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g\" (UID: \"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" Jul 23 00:41:09.244452 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.244422 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5mh2\" (UniqueName: \"kubernetes.io/projected/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-kube-api-access-s5mh2\") pod \"maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g\" (UID: \"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" Jul 23 00:41:09.634537 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.634500 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:09.735710 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.735681 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:09.738773 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.738755 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-maas-api-tls\") pod \"maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g\" (UID: \"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" Jul 23 00:41:09.741091 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.741075 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-maas-api-tls\") pod \"maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g\" (UID: \"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" Jul 23 00:41:09.967941 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:09.967901 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" Jul 23 00:41:10.104563 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:10.104537 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g"] Jul 23 00:41:10.106837 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:41:10.106813 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc8664d_7582_4ffa_bcc8_7c470bd6d17e.slice/crio-56dd0d9fe0cdcb20b143de422b36d5abada48eda217199b3d380b39268de4b3d WatchSource:0}: Error finding container 56dd0d9fe0cdcb20b143de422b36d5abada48eda217199b3d380b39268de4b3d: Status 404 returned error can't find the container with id 56dd0d9fe0cdcb20b143de422b36d5abada48eda217199b3d380b39268de4b3d Jul 23 00:41:10.296104 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:10.296016 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" event={"ID":"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e","Type":"ContainerStarted","Data":"56dd0d9fe0cdcb20b143de422b36d5abada48eda217199b3d380b39268de4b3d"} Jul 23 00:41:11.301533 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:11.301499 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" event={"ID":"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e","Type":"ContainerStarted","Data":"642068f34dbef90617c4517cf51fd4e4dc0c188bf12860b56bc920807fba665e"} Jul 23 00:41:11.302013 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:11.301607 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" Jul 23 00:41:11.317199 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:11.317155 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" podStartSLOduration=2.045021287 podStartE2EDuration="2.317142615s" totalImagesPullingTime="272.121328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:41:11.316205151 +0000 UTC m=+1991.276943769" watchObservedRunningTime="2026-07-23 00:41:11.317142615 +0000 UTC m=+1991.277881234" Jul 23 00:41:17.313512 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:17.313482 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" Jul 23 00:41:44.301692 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:44.301664 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g"] Jul 23 00:41:44.302236 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:44.301889 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" podUID="cbc8664d-7582-4ffa-bcc8-7c470bd6d17e" containerName="maas-api" containerID="cri-o://642068f34dbef90617c4517cf51fd4e4dc0c188bf12860b56bc920807fba665e" gracePeriod=30 Jul 23 00:41:44.327713 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:44.327685 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:44.459732 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:44.459698 2555 generic.go:356] "Generic (PLEG): container finished" podID="cbc8664d-7582-4ffa-bcc8-7c470bd6d17e" containerID="642068f34dbef90617c4517cf51fd4e4dc0c188bf12860b56bc920807fba665e" exitCode=0 Jul 23 00:41:44.459886 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:44.459772 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" event={"ID":"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e","Type":"ContainerDied","Data":"642068f34dbef90617c4517cf51fd4e4dc0c188bf12860b56bc920807fba665e"} Jul 23 00:41:44.550644 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:44.550624 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" Jul 23 00:41:44.651605 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:44.651523 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-maas-api-tls\") pod \"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e\" (UID: \"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e\") " Jul 23 00:41:44.651764 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:44.651617 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-openshift-service-ca\") pod \"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e\" (UID: \"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e\") " Jul 23 00:41:44.651764 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:44.651654 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5mh2\" (UniqueName: \"kubernetes.io/projected/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-kube-api-access-s5mh2\") pod \"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e\" (UID: \"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e\") " Jul 23 00:41:44.651986 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:44.651962 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cbc8664d-7582-4ffa-bcc8-7c470bd6d17e" (UID: "cbc8664d-7582-4ffa-bcc8-7c470bd6d17e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:41:44.653720 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:44.653695 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cbc8664d-7582-4ffa-bcc8-7c470bd6d17e" (UID: "cbc8664d-7582-4ffa-bcc8-7c470bd6d17e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:41:44.653808 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:44.653700 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-kube-api-access-s5mh2" (OuterVolumeSpecName: "kube-api-access-s5mh2") pod "cbc8664d-7582-4ffa-bcc8-7c470bd6d17e" (UID: "cbc8664d-7582-4ffa-bcc8-7c470bd6d17e"). InnerVolumeSpecName "kube-api-access-s5mh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:41:44.752969 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:44.752936 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-openshift-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:41:44.752969 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:44.752964 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5mh2\" (UniqueName: \"kubernetes.io/projected/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-kube-api-access-s5mh2\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:41:44.752969 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:44.752974 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e-maas-api-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:41:45.465526 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:45.465492 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" event={"ID":"cbc8664d-7582-4ffa-bcc8-7c470bd6d17e","Type":"ContainerDied","Data":"56dd0d9fe0cdcb20b143de422b36d5abada48eda217199b3d380b39268de4b3d"} Jul 23 00:41:45.465526 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:45.465527 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g" Jul 23 00:41:45.466036 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:45.465552 2555 scope.go:117] "RemoveContainer" containerID="642068f34dbef90617c4517cf51fd4e4dc0c188bf12860b56bc920807fba665e" Jul 23 00:41:45.486362 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:45.486337 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g"] Jul 23 00:41:45.489842 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:45.489822 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-72954db1-58b89c7b77-2zv5g"] Jul 23 00:41:46.502551 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:46.502511 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbc8664d-7582-4ffa-bcc8-7c470bd6d17e" path="/var/lib/kubelet/pods/cbc8664d-7582-4ffa-bcc8-7c470bd6d17e/volumes" Jul 23 00:41:48.993101 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:48.993069 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:49.157646 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:49.157610 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:49.417085 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:49.417029 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:49.668654 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:49.668571 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:49.994692 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:49.994611 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:50.447736 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:50.447698 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:52.478309 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.478272 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:52.634862 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.634830 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:52.728794 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.728724 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw"] Jul 23 00:41:52.729072 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.729060 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc8664d-7582-4ffa-bcc8-7c470bd6d17e" containerName="maas-api" Jul 23 00:41:52.729116 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.729073 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc8664d-7582-4ffa-bcc8-7c470bd6d17e" containerName="maas-api" Jul 23 00:41:52.729174 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.729165 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc8664d-7582-4ffa-bcc8-7c470bd6d17e" containerName="maas-api" Jul 23 00:41:52.733648 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.733631 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" Jul 23 00:41:52.736583 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.736560 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d0be4a18-serving-cert\"" Jul 23 00:41:52.736706 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.736664 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gktdb\"" Jul 23 00:41:52.741615 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.741592 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw"] Jul 23 00:41:52.758876 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.758848 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:52.827116 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.827084 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32b54e2-88c5-446f-b0e0-cff45c2587e7-openshift-service-ca\") pod \"maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw\" (UID: \"c32b54e2-88c5-446f-b0e0-cff45c2587e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" Jul 23 00:41:52.827297 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.827127 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mr8s\" (UniqueName: \"kubernetes.io/projected/c32b54e2-88c5-446f-b0e0-cff45c2587e7-kube-api-access-4mr8s\") pod \"maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw\" (UID: \"c32b54e2-88c5-446f-b0e0-cff45c2587e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" Jul 23 00:41:52.827297 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.827199 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c32b54e2-88c5-446f-b0e0-cff45c2587e7-maas-api-tls\") pod \"maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw\" (UID: \"c32b54e2-88c5-446f-b0e0-cff45c2587e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" Jul 23 00:41:52.928500 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.928473 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c32b54e2-88c5-446f-b0e0-cff45c2587e7-maas-api-tls\") pod \"maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw\" (UID: \"c32b54e2-88c5-446f-b0e0-cff45c2587e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" Jul 23 00:41:52.928674 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.928529 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32b54e2-88c5-446f-b0e0-cff45c2587e7-openshift-service-ca\") pod \"maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw\" (UID: \"c32b54e2-88c5-446f-b0e0-cff45c2587e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" Jul 23 00:41:52.928674 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.928554 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mr8s\" (UniqueName: \"kubernetes.io/projected/c32b54e2-88c5-446f-b0e0-cff45c2587e7-kube-api-access-4mr8s\") pod \"maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw\" (UID: \"c32b54e2-88c5-446f-b0e0-cff45c2587e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" Jul 23 00:41:52.928809 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:41:52.928700 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-serving-cert: secret "maas-api-e2e-ait-d0be4a18-serving-cert" not found Jul 23 00:41:52.928809 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:41:52.928782 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c32b54e2-88c5-446f-b0e0-cff45c2587e7-maas-api-tls podName:c32b54e2-88c5-446f-b0e0-cff45c2587e7 nodeName:}" failed. No retries permitted until 2026-07-23 00:41:53.428760865 +0000 UTC m=+2033.389499469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c32b54e2-88c5-446f-b0e0-cff45c2587e7-maas-api-tls") pod "maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" (UID: "c32b54e2-88c5-446f-b0e0-cff45c2587e7") : secret "maas-api-e2e-ait-d0be4a18-serving-cert" not found Jul 23 00:41:52.929194 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.929172 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32b54e2-88c5-446f-b0e0-cff45c2587e7-openshift-service-ca\") pod \"maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw\" (UID: \"c32b54e2-88c5-446f-b0e0-cff45c2587e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" Jul 23 00:41:52.936843 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:52.936821 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mr8s\" (UniqueName: \"kubernetes.io/projected/c32b54e2-88c5-446f-b0e0-cff45c2587e7-kube-api-access-4mr8s\") pod \"maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw\" (UID: \"c32b54e2-88c5-446f-b0e0-cff45c2587e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" Jul 23 00:41:53.253605 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:53.253570 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:53.356808 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:53.356777 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:53.433850 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:53.433820 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c32b54e2-88c5-446f-b0e0-cff45c2587e7-maas-api-tls\") pod \"maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw\" (UID: \"c32b54e2-88c5-446f-b0e0-cff45c2587e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" Jul 23 00:41:53.436184 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:53.436162 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c32b54e2-88c5-446f-b0e0-cff45c2587e7-maas-api-tls\") pod \"maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw\" (UID: \"c32b54e2-88c5-446f-b0e0-cff45c2587e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" Jul 23 00:41:53.461691 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:53.461667 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:53.646198 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:53.646122 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" Jul 23 00:41:53.784508 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:53.784485 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw"] Jul 23 00:41:53.786791 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:41:53.786759 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32b54e2_88c5_446f_b0e0_cff45c2587e7.slice/crio-763fd516c5646cff726ff6839c92ecd9236ce19213b63de07e6bd0d40bedc90f WatchSource:0}: Error finding container 763fd516c5646cff726ff6839c92ecd9236ce19213b63de07e6bd0d40bedc90f: Status 404 returned error can't find the container with id 763fd516c5646cff726ff6839c92ecd9236ce19213b63de07e6bd0d40bedc90f Jul 23 00:41:54.509151 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:54.509113 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" event={"ID":"c32b54e2-88c5-446f-b0e0-cff45c2587e7","Type":"ContainerStarted","Data":"f8d3ab768f40f4de1a07699bb578f63929923fcf49e78107447c8c9210043d7b"} Jul 23 00:41:54.509263 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:54.509167 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" event={"ID":"c32b54e2-88c5-446f-b0e0-cff45c2587e7","Type":"ContainerStarted","Data":"763fd516c5646cff726ff6839c92ecd9236ce19213b63de07e6bd0d40bedc90f"} Jul 23 00:41:54.509263 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:54.509255 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" Jul 23 00:41:54.525739 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:41:54.525700 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" podStartSLOduration=1.861975829 podStartE2EDuration="2.525687209s" totalImagesPullingTime="663.71138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:41:54.525126784 +0000 UTC m=+2034.485865409" watchObservedRunningTime="2026-07-23 00:41:54.525687209 +0000 UTC m=+2034.486425829" Jul 23 00:42:00.519205 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:00.519177 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" Jul 23 00:42:14.556081 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:14.556049 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:14.658081 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:14.658048 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:15.258952 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:15.258922 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:15.360818 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:15.360792 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:28.891918 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:28.891886 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw"] Jul 23 00:42:28.892375 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:28.892107 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" podUID="c32b54e2-88c5-446f-b0e0-cff45c2587e7" containerName="maas-api" containerID="cri-o://f8d3ab768f40f4de1a07699bb578f63929923fcf49e78107447c8c9210043d7b" gracePeriod=30 Jul 23 00:42:28.916676 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:28.916653 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:29.140192 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.140170 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" Jul 23 00:42:29.249959 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.249927 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mr8s\" (UniqueName: \"kubernetes.io/projected/c32b54e2-88c5-446f-b0e0-cff45c2587e7-kube-api-access-4mr8s\") pod \"c32b54e2-88c5-446f-b0e0-cff45c2587e7\" (UID: \"c32b54e2-88c5-446f-b0e0-cff45c2587e7\") " Jul 23 00:42:29.250165 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.249995 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32b54e2-88c5-446f-b0e0-cff45c2587e7-openshift-service-ca\") pod \"c32b54e2-88c5-446f-b0e0-cff45c2587e7\" (UID: \"c32b54e2-88c5-446f-b0e0-cff45c2587e7\") " Jul 23 00:42:29.250165 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.250012 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c32b54e2-88c5-446f-b0e0-cff45c2587e7-maas-api-tls\") pod \"c32b54e2-88c5-446f-b0e0-cff45c2587e7\" (UID: \"c32b54e2-88c5-446f-b0e0-cff45c2587e7\") " Jul 23 00:42:29.250442 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.250386 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32b54e2-88c5-446f-b0e0-cff45c2587e7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c32b54e2-88c5-446f-b0e0-cff45c2587e7" (UID: "c32b54e2-88c5-446f-b0e0-cff45c2587e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:42:29.252138 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.252108 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c32b54e2-88c5-446f-b0e0-cff45c2587e7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c32b54e2-88c5-446f-b0e0-cff45c2587e7" (UID: "c32b54e2-88c5-446f-b0e0-cff45c2587e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:42:29.252236 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.252176 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c32b54e2-88c5-446f-b0e0-cff45c2587e7-kube-api-access-4mr8s" (OuterVolumeSpecName: "kube-api-access-4mr8s") pod "c32b54e2-88c5-446f-b0e0-cff45c2587e7" (UID: "c32b54e2-88c5-446f-b0e0-cff45c2587e7"). InnerVolumeSpecName "kube-api-access-4mr8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:42:29.351204 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.351172 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mr8s\" (UniqueName: \"kubernetes.io/projected/c32b54e2-88c5-446f-b0e0-cff45c2587e7-kube-api-access-4mr8s\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:42:29.351204 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.351199 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32b54e2-88c5-446f-b0e0-cff45c2587e7-openshift-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:42:29.351204 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.351209 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c32b54e2-88c5-446f-b0e0-cff45c2587e7-maas-api-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:42:29.680360 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.680331 2555 generic.go:356] "Generic (PLEG): container finished" podID="c32b54e2-88c5-446f-b0e0-cff45c2587e7" containerID="f8d3ab768f40f4de1a07699bb578f63929923fcf49e78107447c8c9210043d7b" exitCode=0 Jul 23 00:42:29.680543 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.680374 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" Jul 23 00:42:29.680543 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.680385 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" event={"ID":"c32b54e2-88c5-446f-b0e0-cff45c2587e7","Type":"ContainerDied","Data":"f8d3ab768f40f4de1a07699bb578f63929923fcf49e78107447c8c9210043d7b"} Jul 23 00:42:29.680543 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.680449 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw" event={"ID":"c32b54e2-88c5-446f-b0e0-cff45c2587e7","Type":"ContainerDied","Data":"763fd516c5646cff726ff6839c92ecd9236ce19213b63de07e6bd0d40bedc90f"} Jul 23 00:42:29.680543 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.680470 2555 scope.go:117] "RemoveContainer" containerID="f8d3ab768f40f4de1a07699bb578f63929923fcf49e78107447c8c9210043d7b" Jul 23 00:42:29.688971 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.688954 2555 scope.go:117] "RemoveContainer" containerID="f8d3ab768f40f4de1a07699bb578f63929923fcf49e78107447c8c9210043d7b" Jul 23 00:42:29.689211 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:42:29.689191 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d3ab768f40f4de1a07699bb578f63929923fcf49e78107447c8c9210043d7b\": container with ID starting with f8d3ab768f40f4de1a07699bb578f63929923fcf49e78107447c8c9210043d7b not found: ID does not exist" containerID="f8d3ab768f40f4de1a07699bb578f63929923fcf49e78107447c8c9210043d7b" Jul 23 00:42:29.689250 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.689220 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d3ab768f40f4de1a07699bb578f63929923fcf49e78107447c8c9210043d7b"} err="failed to get container status \"f8d3ab768f40f4de1a07699bb578f63929923fcf49e78107447c8c9210043d7b\": rpc error: code = NotFound desc = could not find container \"f8d3ab768f40f4de1a07699bb578f63929923fcf49e78107447c8c9210043d7b\": container with ID starting with f8d3ab768f40f4de1a07699bb578f63929923fcf49e78107447c8c9210043d7b not found: ID does not exist" Jul 23 00:42:29.701425 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.701380 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw"] Jul 23 00:42:29.704640 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:29.704612 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0be4a18-7b7c7f867-74jvw"] Jul 23 00:42:30.501821 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:30.501789 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c32b54e2-88c5-446f-b0e0-cff45c2587e7" path="/var/lib/kubelet/pods/c32b54e2-88c5-446f-b0e0-cff45c2587e7/volumes" Jul 23 00:42:33.467308 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:33.467243 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:34.711780 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:34.711748 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:34.955695 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:34.955663 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:35.171869 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:35.171830 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:35.485117 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:35.485001 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:35.883946 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:35.883852 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:37.742455 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:37.742418 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:37.815940 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:37.815892 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:37.820192 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:37.820167 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7"] Jul 23 00:42:37.820697 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:37.820664 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c32b54e2-88c5-446f-b0e0-cff45c2587e7" containerName="maas-api" Jul 23 00:42:37.820697 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:37.820686 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c32b54e2-88c5-446f-b0e0-cff45c2587e7" containerName="maas-api" Jul 23 00:42:37.820860 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:37.820791 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c32b54e2-88c5-446f-b0e0-cff45c2587e7" containerName="maas-api" Jul 23 00:42:37.824041 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:37.824022 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" Jul 23 00:42:37.826686 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:37.826669 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3fad28dc-serving-cert\"" Jul 23 00:42:37.827170 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:37.827153 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gktdb\"" Jul 23 00:42:37.834122 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:37.834100 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7"] Jul 23 00:42:37.891875 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:37.891847 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:37.927825 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:37.927794 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xddrz\" (UniqueName: \"kubernetes.io/projected/52af8fe2-6bfb-470f-a500-adc337953842-kube-api-access-xddrz\") pod \"maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7\" (UID: \"52af8fe2-6bfb-470f-a500-adc337953842\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" Jul 23 00:42:37.927956 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:37.927850 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52af8fe2-6bfb-470f-a500-adc337953842-maas-api-tls\") pod \"maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7\" (UID: \"52af8fe2-6bfb-470f-a500-adc337953842\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" Jul 23 00:42:37.927956 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:37.927903 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52af8fe2-6bfb-470f-a500-adc337953842-openshift-service-ca\") pod \"maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7\" (UID: \"52af8fe2-6bfb-470f-a500-adc337953842\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" Jul 23 00:42:38.008482 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:38.008381 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:38.028995 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:38.028966 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52af8fe2-6bfb-470f-a500-adc337953842-maas-api-tls\") pod \"maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7\" (UID: \"52af8fe2-6bfb-470f-a500-adc337953842\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" Jul 23 00:42:38.029095 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:38.029009 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52af8fe2-6bfb-470f-a500-adc337953842-openshift-service-ca\") pod \"maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7\" (UID: \"52af8fe2-6bfb-470f-a500-adc337953842\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" Jul 23 00:42:38.029095 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:38.029055 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xddrz\" (UniqueName: \"kubernetes.io/projected/52af8fe2-6bfb-470f-a500-adc337953842-kube-api-access-xddrz\") pod \"maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7\" (UID: \"52af8fe2-6bfb-470f-a500-adc337953842\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" Jul 23 00:42:38.029718 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:38.029694 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52af8fe2-6bfb-470f-a500-adc337953842-openshift-service-ca\") pod \"maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7\" (UID: \"52af8fe2-6bfb-470f-a500-adc337953842\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" Jul 23 00:42:38.031456 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:38.031435 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52af8fe2-6bfb-470f-a500-adc337953842-maas-api-tls\") pod \"maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7\" (UID: \"52af8fe2-6bfb-470f-a500-adc337953842\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" Jul 23 00:42:38.041978 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:38.041948 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xddrz\" (UniqueName: \"kubernetes.io/projected/52af8fe2-6bfb-470f-a500-adc337953842-kube-api-access-xddrz\") pod \"maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7\" (UID: \"52af8fe2-6bfb-470f-a500-adc337953842\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" Jul 23 00:42:38.136098 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:38.136062 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" Jul 23 00:42:38.273042 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:38.273017 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7"] Jul 23 00:42:38.275233 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:42:38.275203 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52af8fe2_6bfb_470f_a500_adc337953842.slice/crio-45b3f9c70ec4f5fb33c498a04b92322bb14af6e1381dfd01c69fbe6f585650de WatchSource:0}: Error finding container 45b3f9c70ec4f5fb33c498a04b92322bb14af6e1381dfd01c69fbe6f585650de: Status 404 returned error can't find the container with id 45b3f9c70ec4f5fb33c498a04b92322bb14af6e1381dfd01c69fbe6f585650de Jul 23 00:42:38.499564 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:38.499530 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:38.599652 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:38.599571 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:38.721667 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:38.721627 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" event={"ID":"52af8fe2-6bfb-470f-a500-adc337953842","Type":"ContainerStarted","Data":"45b3f9c70ec4f5fb33c498a04b92322bb14af6e1381dfd01c69fbe6f585650de"} Jul 23 00:42:39.728086 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:39.728051 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" event={"ID":"52af8fe2-6bfb-470f-a500-adc337953842","Type":"ContainerStarted","Data":"3dbc0d80e0b91df74d1512b87349789068410d5f4244595ae76ca58eb71c54c0"} Jul 23 00:42:39.728563 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:39.728253 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" Jul 23 00:42:39.743616 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:39.743566 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" podStartSLOduration=1.8206478050000001 podStartE2EDuration="2.743548149s" totalImagesPullingTime="922.900344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:42:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:42:39.74235247 +0000 UTC m=+2079.703091090" watchObservedRunningTime="2026-07-23 00:42:39.743548149 +0000 UTC m=+2079.704286770" Jul 23 00:42:45.739321 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:42:45.739292 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" Jul 23 00:43:00.534989 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:00.534962 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:43:00.543635 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:00.543615 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:43:13.099325 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.099294 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7"] Jul 23 00:43:13.099776 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.099544 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" podUID="52af8fe2-6bfb-470f-a500-adc337953842" containerName="maas-api" containerID="cri-o://3dbc0d80e0b91df74d1512b87349789068410d5f4244595ae76ca58eb71c54c0" gracePeriod=30 Jul 23 00:43:13.128522 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.128492 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:43:13.349643 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.349567 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" Jul 23 00:43:13.422667 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.422635 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52af8fe2-6bfb-470f-a500-adc337953842-openshift-service-ca\") pod \"52af8fe2-6bfb-470f-a500-adc337953842\" (UID: \"52af8fe2-6bfb-470f-a500-adc337953842\") " Jul 23 00:43:13.422857 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.422713 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xddrz\" (UniqueName: \"kubernetes.io/projected/52af8fe2-6bfb-470f-a500-adc337953842-kube-api-access-xddrz\") pod \"52af8fe2-6bfb-470f-a500-adc337953842\" (UID: \"52af8fe2-6bfb-470f-a500-adc337953842\") " Jul 23 00:43:13.422857 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.422746 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52af8fe2-6bfb-470f-a500-adc337953842-maas-api-tls\") pod \"52af8fe2-6bfb-470f-a500-adc337953842\" (UID: \"52af8fe2-6bfb-470f-a500-adc337953842\") " Jul 23 00:43:13.423021 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.422994 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52af8fe2-6bfb-470f-a500-adc337953842-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52af8fe2-6bfb-470f-a500-adc337953842" (UID: "52af8fe2-6bfb-470f-a500-adc337953842"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:43:13.424836 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.424812 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52af8fe2-6bfb-470f-a500-adc337953842-kube-api-access-xddrz" (OuterVolumeSpecName: "kube-api-access-xddrz") pod "52af8fe2-6bfb-470f-a500-adc337953842" (UID: "52af8fe2-6bfb-470f-a500-adc337953842"). InnerVolumeSpecName "kube-api-access-xddrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:43:13.424931 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.424856 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52af8fe2-6bfb-470f-a500-adc337953842-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52af8fe2-6bfb-470f-a500-adc337953842" (UID: "52af8fe2-6bfb-470f-a500-adc337953842"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:43:13.523660 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.523617 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52af8fe2-6bfb-470f-a500-adc337953842-openshift-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:43:13.523660 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.523648 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xddrz\" (UniqueName: \"kubernetes.io/projected/52af8fe2-6bfb-470f-a500-adc337953842-kube-api-access-xddrz\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:43:13.523660 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.523659 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52af8fe2-6bfb-470f-a500-adc337953842-maas-api-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:43:13.892586 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.892547 2555 generic.go:356] "Generic (PLEG): container finished" podID="52af8fe2-6bfb-470f-a500-adc337953842" containerID="3dbc0d80e0b91df74d1512b87349789068410d5f4244595ae76ca58eb71c54c0" exitCode=0 Jul 23 00:43:13.892586 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.892589 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" Jul 23 00:43:13.892809 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.892640 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" event={"ID":"52af8fe2-6bfb-470f-a500-adc337953842","Type":"ContainerDied","Data":"3dbc0d80e0b91df74d1512b87349789068410d5f4244595ae76ca58eb71c54c0"} Jul 23 00:43:13.892809 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.892695 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7" event={"ID":"52af8fe2-6bfb-470f-a500-adc337953842","Type":"ContainerDied","Data":"45b3f9c70ec4f5fb33c498a04b92322bb14af6e1381dfd01c69fbe6f585650de"} Jul 23 00:43:13.892809 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.892723 2555 scope.go:117] "RemoveContainer" containerID="3dbc0d80e0b91df74d1512b87349789068410d5f4244595ae76ca58eb71c54c0" Jul 23 00:43:13.902045 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.902028 2555 scope.go:117] "RemoveContainer" containerID="3dbc0d80e0b91df74d1512b87349789068410d5f4244595ae76ca58eb71c54c0" Jul 23 00:43:13.902321 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:43:13.902287 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dbc0d80e0b91df74d1512b87349789068410d5f4244595ae76ca58eb71c54c0\": container with ID starting with 3dbc0d80e0b91df74d1512b87349789068410d5f4244595ae76ca58eb71c54c0 not found: ID does not exist" containerID="3dbc0d80e0b91df74d1512b87349789068410d5f4244595ae76ca58eb71c54c0" Jul 23 00:43:13.902558 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.902333 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbc0d80e0b91df74d1512b87349789068410d5f4244595ae76ca58eb71c54c0"} err="failed to get container status \"3dbc0d80e0b91df74d1512b87349789068410d5f4244595ae76ca58eb71c54c0\": rpc error: code = NotFound desc = could not find container \"3dbc0d80e0b91df74d1512b87349789068410d5f4244595ae76ca58eb71c54c0\": container with ID starting with 3dbc0d80e0b91df74d1512b87349789068410d5f4244595ae76ca58eb71c54c0 not found: ID does not exist" Jul 23 00:43:13.914065 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.914042 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7"] Jul 23 00:43:13.917084 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:13.917060 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3fad28dc-5b785c9474-kbpz7"] Jul 23 00:43:14.502685 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:14.502647 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52af8fe2-6bfb-470f-a500-adc337953842" path="/var/lib/kubelet/pods/52af8fe2-6bfb-470f-a500-adc337953842/volumes" Jul 23 00:43:17.726201 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:17.726151 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:43:17.884117 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:17.884085 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:43:18.889089 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:18.889058 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:43:18.989052 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:18.989022 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:43:20.324403 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:20.324370 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:43:26.437343 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:26.437294 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:43:26.541548 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:26.541507 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:43:26.640379 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:26.640325 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:43:26.737426 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:26.737305 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:43:32.297481 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:32.297444 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:43:58.315011 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:43:58.314916 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:04.496590 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:04.496555 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:04.599786 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:04.599753 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:04.913142 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:04.913107 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:05.012038 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:05.012006 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:21.075702 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:21.075654 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:26.749402 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:26.749368 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:26.810237 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:26.810208 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:26.874311 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:26.874277 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:26.974980 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:26.974925 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:37.481663 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:37.481627 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:38.293149 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:38.293114 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:44.901691 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:44.901615 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:45.008540 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:45.008502 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:45.106699 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:45.106660 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:45.211551 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:45.211522 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:45.307338 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:45.307311 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:46.154324 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:46.154285 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:46.246898 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:46.246866 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:48.050546 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:48.050518 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:54.033175 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:54.033136 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:54.134626 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:54.134594 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:44:56.422139 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:44:56.422101 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:00.716035 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:00.715997 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:00.816933 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:00.816895 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:00.957825 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:00.957789 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:01.270968 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:01.270935 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:01.937149 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:01.937101 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:04.173854 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.173813 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:04.245437 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.245382 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt"] Jul 23 00:45:04.245823 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.245808 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52af8fe2-6bfb-470f-a500-adc337953842" containerName="maas-api" Jul 23 00:45:04.245823 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.245824 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="52af8fe2-6bfb-470f-a500-adc337953842" containerName="maas-api" Jul 23 00:45:04.245940 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.245894 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="52af8fe2-6bfb-470f-a500-adc337953842" containerName="maas-api" Jul 23 00:45:04.249002 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.248981 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" Jul 23 00:45:04.252676 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.252652 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c65690b4-serving-cert\"" Jul 23 00:45:04.252676 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.252669 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gktdb\"" Jul 23 00:45:04.261871 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.261846 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt"] Jul 23 00:45:04.296898 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.296871 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:04.323336 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.323292 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfw5b\" (UniqueName: \"kubernetes.io/projected/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-kube-api-access-sfw5b\") pod \"maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt\" (UID: \"1461d455-b5a2-49ff-ac65-cce01ab5b2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" Jul 23 00:45:04.323528 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.323344 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-openshift-service-ca\") pod \"maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt\" (UID: \"1461d455-b5a2-49ff-ac65-cce01ab5b2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" Jul 23 00:45:04.323528 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.323501 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-maas-api-tls\") pod \"maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt\" (UID: \"1461d455-b5a2-49ff-ac65-cce01ab5b2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" Jul 23 00:45:04.397101 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.397069 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:04.424426 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.424321 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfw5b\" (UniqueName: \"kubernetes.io/projected/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-kube-api-access-sfw5b\") pod \"maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt\" (UID: \"1461d455-b5a2-49ff-ac65-cce01ab5b2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" Jul 23 00:45:04.424426 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.424362 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-openshift-service-ca\") pod \"maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt\" (UID: \"1461d455-b5a2-49ff-ac65-cce01ab5b2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" Jul 23 00:45:04.424661 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.424490 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-maas-api-tls\") pod \"maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt\" (UID: \"1461d455-b5a2-49ff-ac65-cce01ab5b2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" Jul 23 00:45:04.424661 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:45:04.424605 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-serving-cert: secret "maas-api-e2e-mt-c65690b4-serving-cert" not found Jul 23 00:45:04.424778 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:45:04.424677 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-maas-api-tls podName:1461d455-b5a2-49ff-ac65-cce01ab5b2c6 nodeName:}" failed. No retries permitted until 2026-07-23 00:45:04.924657796 +0000 UTC m=+2224.885396418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-maas-api-tls") pod "maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" (UID: "1461d455-b5a2-49ff-ac65-cce01ab5b2c6") : secret "maas-api-e2e-mt-c65690b4-serving-cert" not found Jul 23 00:45:04.425151 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.425126 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-openshift-service-ca\") pod \"maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt\" (UID: \"1461d455-b5a2-49ff-ac65-cce01ab5b2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" Jul 23 00:45:04.436845 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.436813 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfw5b\" (UniqueName: \"kubernetes.io/projected/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-kube-api-access-sfw5b\") pod \"maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt\" (UID: \"1461d455-b5a2-49ff-ac65-cce01ab5b2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" Jul 23 00:45:04.589004 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.588968 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:04.928805 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.928767 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-maas-api-tls\") pod \"maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt\" (UID: \"1461d455-b5a2-49ff-ac65-cce01ab5b2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" Jul 23 00:45:04.931227 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:04.931204 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-maas-api-tls\") pod \"maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt\" (UID: \"1461d455-b5a2-49ff-ac65-cce01ab5b2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" Jul 23 00:45:05.092047 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:05.092014 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:05.160754 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:05.160706 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" Jul 23 00:45:05.194314 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:05.194231 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:05.296300 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:05.296274 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt"] Jul 23 00:45:05.298597 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:45:05.298554 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1461d455_b5a2_49ff_ac65_cce01ab5b2c6.slice/crio-d693616aa12008a72fbc7ed1d9874ff9f5d4aaae4671f605e588a8c117c8a31d WatchSource:0}: Error finding container d693616aa12008a72fbc7ed1d9874ff9f5d4aaae4671f605e588a8c117c8a31d: Status 404 returned error can't find the container with id d693616aa12008a72fbc7ed1d9874ff9f5d4aaae4671f605e588a8c117c8a31d Jul 23 00:45:05.424138 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:05.424105 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" event={"ID":"1461d455-b5a2-49ff-ac65-cce01ab5b2c6","Type":"ContainerStarted","Data":"d693616aa12008a72fbc7ed1d9874ff9f5d4aaae4671f605e588a8c117c8a31d"} Jul 23 00:45:06.430465 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:06.430373 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" event={"ID":"1461d455-b5a2-49ff-ac65-cce01ab5b2c6","Type":"ContainerStarted","Data":"4e47040b9f300bccbf2c8983580f117a28b185f0296abf93aff771bf97799e7b"} Jul 23 00:45:06.430855 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:06.430515 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" Jul 23 00:45:06.451339 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:06.451284 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" podStartSLOduration=1.737291827 podStartE2EDuration="2.45126782s" totalImagesPullingTime="713.975993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:45:06.449288423 +0000 UTC m=+2226.410027043" watchObservedRunningTime="2026-07-23 00:45:06.45126782 +0000 UTC m=+2226.412006439" Jul 23 00:45:08.691326 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:08.691291 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:09.382152 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:09.382113 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:09.476664 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:09.476634 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:09.824992 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:09.824959 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:12.441498 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:12.441473 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" Jul 23 00:45:15.473816 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:15.473759 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:15.541537 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:15.541488 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:26.212508 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.212473 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt"] Jul 23 00:45:26.212967 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.212752 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" podUID="1461d455-b5a2-49ff-ac65-cce01ab5b2c6" containerName="maas-api" containerID="cri-o://4e47040b9f300bccbf2c8983580f117a28b185f0296abf93aff771bf97799e7b" gracePeriod=30 Jul 23 00:45:26.232563 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.232532 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:26.459787 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.459764 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" Jul 23 00:45:26.528330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.528250 2555 generic.go:356] "Generic (PLEG): container finished" podID="1461d455-b5a2-49ff-ac65-cce01ab5b2c6" containerID="4e47040b9f300bccbf2c8983580f117a28b185f0296abf93aff771bf97799e7b" exitCode=0 Jul 23 00:45:26.528330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.528301 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" Jul 23 00:45:26.528514 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.528333 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" event={"ID":"1461d455-b5a2-49ff-ac65-cce01ab5b2c6","Type":"ContainerDied","Data":"4e47040b9f300bccbf2c8983580f117a28b185f0296abf93aff771bf97799e7b"} Jul 23 00:45:26.528514 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.528385 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt" event={"ID":"1461d455-b5a2-49ff-ac65-cce01ab5b2c6","Type":"ContainerDied","Data":"d693616aa12008a72fbc7ed1d9874ff9f5d4aaae4671f605e588a8c117c8a31d"} Jul 23 00:45:26.528514 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.528430 2555 scope.go:117] "RemoveContainer" containerID="4e47040b9f300bccbf2c8983580f117a28b185f0296abf93aff771bf97799e7b" Jul 23 00:45:26.536902 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.536886 2555 scope.go:117] "RemoveContainer" containerID="4e47040b9f300bccbf2c8983580f117a28b185f0296abf93aff771bf97799e7b" Jul 23 00:45:26.537152 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:45:26.537130 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e47040b9f300bccbf2c8983580f117a28b185f0296abf93aff771bf97799e7b\": container with ID starting with 4e47040b9f300bccbf2c8983580f117a28b185f0296abf93aff771bf97799e7b not found: ID does not exist" containerID="4e47040b9f300bccbf2c8983580f117a28b185f0296abf93aff771bf97799e7b" Jul 23 00:45:26.537206 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.537160 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e47040b9f300bccbf2c8983580f117a28b185f0296abf93aff771bf97799e7b"} err="failed to get container status \"4e47040b9f300bccbf2c8983580f117a28b185f0296abf93aff771bf97799e7b\": rpc error: code = NotFound desc = could not find container \"4e47040b9f300bccbf2c8983580f117a28b185f0296abf93aff771bf97799e7b\": container with ID starting with 4e47040b9f300bccbf2c8983580f117a28b185f0296abf93aff771bf97799e7b not found: ID does not exist" Jul 23 00:45:26.618068 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.618037 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfw5b\" (UniqueName: \"kubernetes.io/projected/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-kube-api-access-sfw5b\") pod \"1461d455-b5a2-49ff-ac65-cce01ab5b2c6\" (UID: \"1461d455-b5a2-49ff-ac65-cce01ab5b2c6\") " Jul 23 00:45:26.618212 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.618148 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-openshift-service-ca\") pod \"1461d455-b5a2-49ff-ac65-cce01ab5b2c6\" (UID: \"1461d455-b5a2-49ff-ac65-cce01ab5b2c6\") " Jul 23 00:45:26.618212 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.618187 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-maas-api-tls\") pod \"1461d455-b5a2-49ff-ac65-cce01ab5b2c6\" (UID: \"1461d455-b5a2-49ff-ac65-cce01ab5b2c6\") " Jul 23 00:45:26.618589 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.618563 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1461d455-b5a2-49ff-ac65-cce01ab5b2c6" (UID: "1461d455-b5a2-49ff-ac65-cce01ab5b2c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:45:26.620278 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.620255 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1461d455-b5a2-49ff-ac65-cce01ab5b2c6" (UID: "1461d455-b5a2-49ff-ac65-cce01ab5b2c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:45:26.620345 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.620323 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-kube-api-access-sfw5b" (OuterVolumeSpecName: "kube-api-access-sfw5b") pod "1461d455-b5a2-49ff-ac65-cce01ab5b2c6" (UID: "1461d455-b5a2-49ff-ac65-cce01ab5b2c6"). InnerVolumeSpecName "kube-api-access-sfw5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:45:26.719217 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.719176 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-openshift-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:45:26.719217 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.719208 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-maas-api-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:45:26.719217 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.719220 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfw5b\" (UniqueName: \"kubernetes.io/projected/1461d455-b5a2-49ff-ac65-cce01ab5b2c6-kube-api-access-sfw5b\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:45:26.849567 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.849536 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt"] Jul 23 00:45:26.851102 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:26.851081 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c65690b4-5dcf4c8fd6-v2tzt"] Jul 23 00:45:28.502133 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:28.502102 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1461d455-b5a2-49ff-ac65-cce01ab5b2c6" path="/var/lib/kubelet/pods/1461d455-b5a2-49ff-ac65-cce01ab5b2c6/volumes" Jul 23 00:45:30.903893 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:30.903860 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:31.311349 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:31.311305 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:37.103255 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:37.103221 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:37.204293 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:37.204258 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:37.371754 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:37.371662 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:37.604567 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:37.604537 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:37.725608 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:37.725578 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:37.848328 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:37.848298 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:38.006470 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:38.006348 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:43.574552 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:43.574496 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:43.635007 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:43.634965 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:43.717276 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:43.717241 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:43.812606 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:43.812566 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:43.926125 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:43.926088 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:49.687106 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:49.687077 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:49.789467 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:49.789430 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:56.662065 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:56.662029 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:56.968543 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:56.968453 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:57.249773 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:57.249676 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:57.611003 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:57.610919 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:57.663667 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:57.663636 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:59.514572 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.514540 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:59.592712 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.592675 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:59.596626 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.596597 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d"] Jul 23 00:45:59.597006 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.596990 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1461d455-b5a2-49ff-ac65-cce01ab5b2c6" containerName="maas-api" Jul 23 00:45:59.597006 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.597006 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1461d455-b5a2-49ff-ac65-cce01ab5b2c6" containerName="maas-api" Jul 23 00:45:59.597082 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.597056 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1461d455-b5a2-49ff-ac65-cce01ab5b2c6" containerName="maas-api" Jul 23 00:45:59.600125 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.600109 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" Jul 23 00:45:59.602535 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.602514 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-76cb26bd-serving-cert\"" Jul 23 00:45:59.602655 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.602548 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gktdb\"" Jul 23 00:45:59.610336 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.610309 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d"] Jul 23 00:45:59.660406 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.660361 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:59.697863 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.697818 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661d63b4-3b40-4896-9a90-44b406a06972-openshift-service-ca\") pod \"maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d\" (UID: \"661d63b4-3b40-4896-9a90-44b406a06972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" Jul 23 00:45:59.697863 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.697866 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661d63b4-3b40-4896-9a90-44b406a06972-maas-api-tls\") pod \"maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d\" (UID: \"661d63b4-3b40-4896-9a90-44b406a06972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" Jul 23 00:45:59.698114 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.697893 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpkh7\" (UniqueName: \"kubernetes.io/projected/661d63b4-3b40-4896-9a90-44b406a06972-kube-api-access-qpkh7\") pod \"maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d\" (UID: \"661d63b4-3b40-4896-9a90-44b406a06972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" Jul 23 00:45:59.799525 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.799430 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661d63b4-3b40-4896-9a90-44b406a06972-maas-api-tls\") pod \"maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d\" (UID: \"661d63b4-3b40-4896-9a90-44b406a06972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" Jul 23 00:45:59.799525 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.799480 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpkh7\" (UniqueName: \"kubernetes.io/projected/661d63b4-3b40-4896-9a90-44b406a06972-kube-api-access-qpkh7\") pod \"maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d\" (UID: \"661d63b4-3b40-4896-9a90-44b406a06972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" Jul 23 00:45:59.799779 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:45:59.799593 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-serving-cert: secret "maas-api-e2e-mt-76cb26bd-serving-cert" not found Jul 23 00:45:59.799779 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.799617 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661d63b4-3b40-4896-9a90-44b406a06972-openshift-service-ca\") pod \"maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d\" (UID: \"661d63b4-3b40-4896-9a90-44b406a06972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" Jul 23 00:45:59.799779 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:45:59.799696 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661d63b4-3b40-4896-9a90-44b406a06972-maas-api-tls podName:661d63b4-3b40-4896-9a90-44b406a06972 nodeName:}" failed. No retries permitted until 2026-07-23 00:46:00.299673948 +0000 UTC m=+2280.260412545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/661d63b4-3b40-4896-9a90-44b406a06972-maas-api-tls") pod "maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" (UID: "661d63b4-3b40-4896-9a90-44b406a06972") : secret "maas-api-e2e-mt-76cb26bd-serving-cert" not found Jul 23 00:45:59.800292 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.800272 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661d63b4-3b40-4896-9a90-44b406a06972-openshift-service-ca\") pod \"maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d\" (UID: \"661d63b4-3b40-4896-9a90-44b406a06972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" Jul 23 00:45:59.810777 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:45:59.810752 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpkh7\" (UniqueName: \"kubernetes.io/projected/661d63b4-3b40-4896-9a90-44b406a06972-kube-api-access-qpkh7\") pod \"maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d\" (UID: \"661d63b4-3b40-4896-9a90-44b406a06972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" Jul 23 00:46:00.005440 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:00.005380 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:00.161426 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:00.161372 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:00.304326 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:00.304290 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661d63b4-3b40-4896-9a90-44b406a06972-maas-api-tls\") pod \"maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d\" (UID: \"661d63b4-3b40-4896-9a90-44b406a06972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" Jul 23 00:46:00.306667 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:00.306640 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661d63b4-3b40-4896-9a90-44b406a06972-maas-api-tls\") pod \"maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d\" (UID: \"661d63b4-3b40-4896-9a90-44b406a06972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" Jul 23 00:46:00.308536 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:00.308512 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:00.513547 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:00.513475 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gktdb\"" Jul 23 00:46:00.522275 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:00.522252 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" Jul 23 00:46:00.653451 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:00.653429 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d"] Jul 23 00:46:00.655504 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:46:00.655479 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661d63b4_3b40_4896_9a90_44b406a06972.slice/crio-509df80a539d9b1fe16b909d0188983ca3b19c49af0227da597465e90aebb0dd WatchSource:0}: Error finding container 509df80a539d9b1fe16b909d0188983ca3b19c49af0227da597465e90aebb0dd: Status 404 returned error can't find the container with id 509df80a539d9b1fe16b909d0188983ca3b19c49af0227da597465e90aebb0dd Jul 23 00:46:00.656856 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:00.656838 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:46:00.701310 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:00.701280 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" event={"ID":"661d63b4-3b40-4896-9a90-44b406a06972","Type":"ContainerStarted","Data":"509df80a539d9b1fe16b909d0188983ca3b19c49af0227da597465e90aebb0dd"} Jul 23 00:46:03.983253 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:03.983218 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:04.720522 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:04.720485 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" event={"ID":"661d63b4-3b40-4896-9a90-44b406a06972","Type":"ContainerStarted","Data":"5de9bf66bb87f2b9c53c11afb0454429a3992fa3efd04f869b1cec06f709ebce"} Jul 23 00:46:04.720694 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:04.720563 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" Jul 23 00:46:04.739459 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:04.739381 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" podStartSLOduration=2.389632927 podStartE2EDuration="5.739362734s" totalImagesPullingTime="3.349729807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:46:04.738012594 +0000 UTC m=+2284.698751212" watchObservedRunningTime="2026-07-23 00:46:04.739362734 +0000 UTC m=+2284.700101352" Jul 23 00:46:10.731677 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:10.731649 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" Jul 23 00:46:15.224119 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.224085 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh"] Jul 23 00:46:15.227278 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.227261 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.229698 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.229677 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-76cb26bd\"/\"kube-root-ca.crt\"" Jul 23 00:46:15.229807 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.229677 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-76cb26bd\"/\"default-dockercfg-6g6rd\"" Jul 23 00:46:15.230613 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.230598 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-76cb26bd\"/\"openshift-service-ca.crt\"" Jul 23 00:46:15.230666 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.230606 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-76cb26bd\"/\"e2e-lifecycle-model-76cb26bd-kserve-self-signed-certs\"" Jul 23 00:46:15.236364 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.236343 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh"] Jul 23 00:46:15.250593 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.250554 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:15.343911 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.343876 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22837470-e593-4c86-92bd-2f14d7e239ac-tls-certs\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.344075 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.343927 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-model-cache\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.344075 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.343987 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g8bb\" (UniqueName: \"kubernetes.io/projected/22837470-e593-4c86-92bd-2f14d7e239ac-kube-api-access-6g8bb\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.344075 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.344042 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-home\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.344075 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.344065 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-dshm\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.344218 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.344113 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-tmp-dir\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.445092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.445055 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-model-cache\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.445092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.445095 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g8bb\" (UniqueName: \"kubernetes.io/projected/22837470-e593-4c86-92bd-2f14d7e239ac-kube-api-access-6g8bb\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.445310 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.445119 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-home\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.445310 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.445143 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-dshm\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.445310 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.445256 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-tmp-dir\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.445501 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.445344 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22837470-e593-4c86-92bd-2f14d7e239ac-tls-certs\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.445560 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.445549 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-model-cache\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.445635 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.445615 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-home\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.445710 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.445688 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-tmp-dir\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.447450 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.447434 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-dshm\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.447824 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.447806 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22837470-e593-4c86-92bd-2f14d7e239ac-tls-certs\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.453107 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.453085 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g8bb\" (UniqueName: \"kubernetes.io/projected/22837470-e593-4c86-92bd-2f14d7e239ac-kube-api-access-6g8bb\") pod \"e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.537817 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.537730 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:15.671022 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.670992 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh"] Jul 23 00:46:15.673165 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:46:15.673131 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22837470_e593_4c86_92bd_2f14d7e239ac.slice/crio-2b7685b9fc6b1c3acc0724aeda4670762ca549f11c45006347678470e3c5dcd9 WatchSource:0}: Error finding container 2b7685b9fc6b1c3acc0724aeda4670762ca549f11c45006347678470e3c5dcd9: Status 404 returned error can't find the container with id 2b7685b9fc6b1c3acc0724aeda4670762ca549f11c45006347678470e3c5dcd9 Jul 23 00:46:15.775311 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.775269 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" event={"ID":"22837470-e593-4c86-92bd-2f14d7e239ac","Type":"ContainerStarted","Data":"b7bcb9ca328aa1397b06ec35ce8fe7f4c861843fbd14667f56d263d0725e198f"} Jul 23 00:46:15.775507 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.775324 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" event={"ID":"22837470-e593-4c86-92bd-2f14d7e239ac","Type":"ContainerStarted","Data":"2b7685b9fc6b1c3acc0724aeda4670762ca549f11c45006347678470e3c5dcd9"} Jul 23 00:46:15.792164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:15.792058 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" podStartSLOduration=0.792039518 podStartE2EDuration="792.039518ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:46:15.790540832 +0000 UTC m=+2295.751279451" watchObservedRunningTime="2026-07-23 00:46:15.792039518 +0000 UTC m=+2295.752778137" Jul 23 00:46:25.538444 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:25.538412 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:25.538444 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:25.538450 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:25.551121 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:25.551085 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:25.831494 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:25.831408 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:46:35.873904 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:35.873857 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:58.028824 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.028730 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d"] Jul 23 00:46:58.029429 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.029038 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" podUID="661d63b4-3b40-4896-9a90-44b406a06972" containerName="maas-api" containerID="cri-o://5de9bf66bb87f2b9c53c11afb0454429a3992fa3efd04f869b1cec06f709ebce" gracePeriod=30 Jul 23 00:46:58.054623 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.054588 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:46:58.283869 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.283799 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" Jul 23 00:46:58.437585 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.437543 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661d63b4-3b40-4896-9a90-44b406a06972-openshift-service-ca\") pod \"661d63b4-3b40-4896-9a90-44b406a06972\" (UID: \"661d63b4-3b40-4896-9a90-44b406a06972\") " Jul 23 00:46:58.437796 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.437685 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpkh7\" (UniqueName: \"kubernetes.io/projected/661d63b4-3b40-4896-9a90-44b406a06972-kube-api-access-qpkh7\") pod \"661d63b4-3b40-4896-9a90-44b406a06972\" (UID: \"661d63b4-3b40-4896-9a90-44b406a06972\") " Jul 23 00:46:58.437796 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.437724 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661d63b4-3b40-4896-9a90-44b406a06972-maas-api-tls\") pod \"661d63b4-3b40-4896-9a90-44b406a06972\" (UID: \"661d63b4-3b40-4896-9a90-44b406a06972\") " Jul 23 00:46:58.437944 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.437917 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661d63b4-3b40-4896-9a90-44b406a06972-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "661d63b4-3b40-4896-9a90-44b406a06972" (UID: "661d63b4-3b40-4896-9a90-44b406a06972"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:46:58.438032 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.438009 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661d63b4-3b40-4896-9a90-44b406a06972-openshift-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:46:58.439912 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.439877 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661d63b4-3b40-4896-9a90-44b406a06972-kube-api-access-qpkh7" (OuterVolumeSpecName: "kube-api-access-qpkh7") pod "661d63b4-3b40-4896-9a90-44b406a06972" (UID: "661d63b4-3b40-4896-9a90-44b406a06972"). InnerVolumeSpecName "kube-api-access-qpkh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:46:58.440007 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.439926 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/661d63b4-3b40-4896-9a90-44b406a06972-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "661d63b4-3b40-4896-9a90-44b406a06972" (UID: "661d63b4-3b40-4896-9a90-44b406a06972"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:46:58.539458 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.539352 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpkh7\" (UniqueName: \"kubernetes.io/projected/661d63b4-3b40-4896-9a90-44b406a06972-kube-api-access-qpkh7\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:46:58.539458 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.539381 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661d63b4-3b40-4896-9a90-44b406a06972-maas-api-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:46:58.981845 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.981811 2555 generic.go:356] "Generic (PLEG): container finished" podID="661d63b4-3b40-4896-9a90-44b406a06972" containerID="5de9bf66bb87f2b9c53c11afb0454429a3992fa3efd04f869b1cec06f709ebce" exitCode=0 Jul 23 00:46:58.982058 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.981861 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" Jul 23 00:46:58.982058 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.981896 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" event={"ID":"661d63b4-3b40-4896-9a90-44b406a06972","Type":"ContainerDied","Data":"5de9bf66bb87f2b9c53c11afb0454429a3992fa3efd04f869b1cec06f709ebce"} Jul 23 00:46:58.982058 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.981947 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d" event={"ID":"661d63b4-3b40-4896-9a90-44b406a06972","Type":"ContainerDied","Data":"509df80a539d9b1fe16b909d0188983ca3b19c49af0227da597465e90aebb0dd"} Jul 23 00:46:58.982058 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.981969 2555 scope.go:117] "RemoveContainer" containerID="5de9bf66bb87f2b9c53c11afb0454429a3992fa3efd04f869b1cec06f709ebce" Jul 23 00:46:58.990957 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.990934 2555 scope.go:117] "RemoveContainer" containerID="5de9bf66bb87f2b9c53c11afb0454429a3992fa3efd04f869b1cec06f709ebce" Jul 23 00:46:58.991275 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:46:58.991256 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de9bf66bb87f2b9c53c11afb0454429a3992fa3efd04f869b1cec06f709ebce\": container with ID starting with 5de9bf66bb87f2b9c53c11afb0454429a3992fa3efd04f869b1cec06f709ebce not found: ID does not exist" containerID="5de9bf66bb87f2b9c53c11afb0454429a3992fa3efd04f869b1cec06f709ebce" Jul 23 00:46:58.991317 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.991285 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de9bf66bb87f2b9c53c11afb0454429a3992fa3efd04f869b1cec06f709ebce"} err="failed to get container status \"5de9bf66bb87f2b9c53c11afb0454429a3992fa3efd04f869b1cec06f709ebce\": rpc error: code = NotFound desc = could not find container \"5de9bf66bb87f2b9c53c11afb0454429a3992fa3efd04f869b1cec06f709ebce\": container with ID starting with 5de9bf66bb87f2b9c53c11afb0454429a3992fa3efd04f869b1cec06f709ebce not found: ID does not exist" Jul 23 00:46:58.999435 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:58.999405 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d"] Jul 23 00:46:59.011787 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:59.011760 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76cb26bd-66759c997c-h8p8d"] Jul 23 00:46:59.118608 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:46:59.118574 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:00.503189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:00.503157 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="661d63b4-3b40-4896-9a90-44b406a06972" path="/var/lib/kubelet/pods/661d63b4-3b40-4896-9a90-44b406a06972/volumes" Jul 23 00:47:02.692327 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:02.692259 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:06.336026 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:06.335972 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:06.406311 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:06.406263 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:09.994956 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:09.994914 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh"] Jul 23 00:47:10.090959 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:10.090920 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:15.273524 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:15.273493 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:15.389219 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:15.389193 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:25.015658 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.015617 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" podUID="22837470-e593-4c86-92bd-2f14d7e239ac" containerName="main" containerID="cri-o://b7bcb9ca328aa1397b06ec35ce8fe7f4c861843fbd14667f56d263d0725e198f" gracePeriod=45 Jul 23 00:47:25.263883 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.263860 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:47:25.368511 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.368431 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-tmp-dir\") pod \"22837470-e593-4c86-92bd-2f14d7e239ac\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " Jul 23 00:47:25.368511 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.368484 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22837470-e593-4c86-92bd-2f14d7e239ac-tls-certs\") pod \"22837470-e593-4c86-92bd-2f14d7e239ac\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " Jul 23 00:47:25.368511 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.368502 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-dshm\") pod \"22837470-e593-4c86-92bd-2f14d7e239ac\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " Jul 23 00:47:25.368723 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.368622 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-home\") pod \"22837470-e593-4c86-92bd-2f14d7e239ac\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " Jul 23 00:47:25.368723 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.368702 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g8bb\" (UniqueName: \"kubernetes.io/projected/22837470-e593-4c86-92bd-2f14d7e239ac-kube-api-access-6g8bb\") pod \"22837470-e593-4c86-92bd-2f14d7e239ac\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " Jul 23 00:47:25.368798 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.368735 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-model-cache\") pod \"22837470-e593-4c86-92bd-2f14d7e239ac\" (UID: \"22837470-e593-4c86-92bd-2f14d7e239ac\") " Jul 23 00:47:25.368798 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.368741 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22837470-e593-4c86-92bd-2f14d7e239ac" (UID: "22837470-e593-4c86-92bd-2f14d7e239ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:47:25.368900 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.368874 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-home" (OuterVolumeSpecName: "home") pod "22837470-e593-4c86-92bd-2f14d7e239ac" (UID: "22837470-e593-4c86-92bd-2f14d7e239ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:47:25.369028 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.369002 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "22837470-e593-4c86-92bd-2f14d7e239ac" (UID: "22837470-e593-4c86-92bd-2f14d7e239ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:47:25.369147 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.369084 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-home\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:47:25.369147 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.369101 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-model-cache\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:47:25.369147 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.369112 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-tmp-dir\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:47:25.370770 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.370746 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22837470-e593-4c86-92bd-2f14d7e239ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22837470-e593-4c86-92bd-2f14d7e239ac" (UID: "22837470-e593-4c86-92bd-2f14d7e239ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:47:25.370869 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.370750 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-dshm" (OuterVolumeSpecName: "dshm") pod "22837470-e593-4c86-92bd-2f14d7e239ac" (UID: "22837470-e593-4c86-92bd-2f14d7e239ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:47:25.370869 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.370769 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22837470-e593-4c86-92bd-2f14d7e239ac-kube-api-access-6g8bb" (OuterVolumeSpecName: "kube-api-access-6g8bb") pod "22837470-e593-4c86-92bd-2f14d7e239ac" (UID: "22837470-e593-4c86-92bd-2f14d7e239ac"). InnerVolumeSpecName "kube-api-access-6g8bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:47:25.470056 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.470023 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22837470-e593-4c86-92bd-2f14d7e239ac-tls-certs\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:47:25.470056 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.470051 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22837470-e593-4c86-92bd-2f14d7e239ac-dshm\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:47:25.470056 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:25.470060 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g8bb\" (UniqueName: \"kubernetes.io/projected/22837470-e593-4c86-92bd-2f14d7e239ac-kube-api-access-6g8bb\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:47:26.106826 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:26.106790 2555 generic.go:356] "Generic (PLEG): container finished" podID="22837470-e593-4c86-92bd-2f14d7e239ac" containerID="b7bcb9ca328aa1397b06ec35ce8fe7f4c861843fbd14667f56d263d0725e198f" exitCode=0 Jul 23 00:47:26.107336 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:26.106838 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" Jul 23 00:47:26.107336 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:26.106863 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" event={"ID":"22837470-e593-4c86-92bd-2f14d7e239ac","Type":"ContainerDied","Data":"b7bcb9ca328aa1397b06ec35ce8fe7f4c861843fbd14667f56d263d0725e198f"} Jul 23 00:47:26.107336 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:26.106895 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh" event={"ID":"22837470-e593-4c86-92bd-2f14d7e239ac","Type":"ContainerDied","Data":"2b7685b9fc6b1c3acc0724aeda4670762ca549f11c45006347678470e3c5dcd9"} Jul 23 00:47:26.107336 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:26.106911 2555 scope.go:117] "RemoveContainer" containerID="b7bcb9ca328aa1397b06ec35ce8fe7f4c861843fbd14667f56d263d0725e198f" Jul 23 00:47:26.115732 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:26.115717 2555 scope.go:117] "RemoveContainer" containerID="b7bcb9ca328aa1397b06ec35ce8fe7f4c861843fbd14667f56d263d0725e198f" Jul 23 00:47:26.115971 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:47:26.115953 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7bcb9ca328aa1397b06ec35ce8fe7f4c861843fbd14667f56d263d0725e198f\": container with ID starting with b7bcb9ca328aa1397b06ec35ce8fe7f4c861843fbd14667f56d263d0725e198f not found: ID does not exist" containerID="b7bcb9ca328aa1397b06ec35ce8fe7f4c861843fbd14667f56d263d0725e198f" Jul 23 00:47:26.116030 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:26.115983 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7bcb9ca328aa1397b06ec35ce8fe7f4c861843fbd14667f56d263d0725e198f"} err="failed to get container status \"b7bcb9ca328aa1397b06ec35ce8fe7f4c861843fbd14667f56d263d0725e198f\": rpc error: code = NotFound desc = could not find container \"b7bcb9ca328aa1397b06ec35ce8fe7f4c861843fbd14667f56d263d0725e198f\": container with ID starting with b7bcb9ca328aa1397b06ec35ce8fe7f4c861843fbd14667f56d263d0725e198f not found: ID does not exist" Jul 23 00:47:26.126135 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:26.126112 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh"] Jul 23 00:47:26.128163 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:26.128142 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-76cb26bd/e2e-lifecycle-model-76cb26bd-kserve-99878c7bc-xnbqh"] Jul 23 00:47:26.503404 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:26.503366 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22837470-e593-4c86-92bd-2f14d7e239ac" path="/var/lib/kubelet/pods/22837470-e593-4c86-92bd-2f14d7e239ac/volumes" Jul 23 00:47:34.575143 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:34.575093 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:35.354501 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:35.354467 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:40.870876 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:47:40.870843 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:00.560909 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:00.560879 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:48:00.571040 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:00.571019 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:48:03.451592 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:03.451558 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:03.508319 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:03.508283 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:05.902559 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:05.902526 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:06.000424 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:06.000385 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:06.099920 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:06.099889 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:06.202600 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:06.202567 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:06.300805 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:06.300769 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:07.708021 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:07.707992 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:13.458804 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:13.458768 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:13.518638 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:13.518593 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:14.132311 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:14.132279 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:20.045844 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:20.045808 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:22.081174 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:22.081086 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:22.180916 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:22.180878 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:30.809553 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:30.809514 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:31.164554 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:31.164517 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:52.164808 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:52.164772 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:58.480615 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:58.480581 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:58.578033 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:58.577997 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:58.857140 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:58.857064 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:59.196795 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:59.196762 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:59.481351 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:48:59.481263 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:02.549678 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.549644 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:02.735211 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.735173 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c"] Jul 23 00:49:02.735637 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.735620 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="661d63b4-3b40-4896-9a90-44b406a06972" containerName="maas-api" Jul 23 00:49:02.735637 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.735637 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="661d63b4-3b40-4896-9a90-44b406a06972" containerName="maas-api" Jul 23 00:49:02.735797 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.735652 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22837470-e593-4c86-92bd-2f14d7e239ac" containerName="main" Jul 23 00:49:02.735797 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.735658 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="22837470-e593-4c86-92bd-2f14d7e239ac" containerName="main" Jul 23 00:49:02.735797 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.735741 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="22837470-e593-4c86-92bd-2f14d7e239ac" containerName="main" Jul 23 00:49:02.735797 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.735753 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="661d63b4-3b40-4896-9a90-44b406a06972" containerName="maas-api" Jul 23 00:49:02.738751 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.738729 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" Jul 23 00:49:02.741186 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.741160 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e5bc11-serving-cert\"" Jul 23 00:49:02.741299 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.741228 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gktdb\"" Jul 23 00:49:02.747887 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.747861 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c"] Jul 23 00:49:02.855470 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.855362 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psrw7\" (UniqueName: \"kubernetes.io/projected/344d7bc1-dd95-406d-a542-ef94377b648b-kube-api-access-psrw7\") pod \"maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c\" (UID: \"344d7bc1-dd95-406d-a542-ef94377b648b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" Jul 23 00:49:02.855470 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.855426 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344d7bc1-dd95-406d-a542-ef94377b648b-maas-api-tls\") pod \"maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c\" (UID: \"344d7bc1-dd95-406d-a542-ef94377b648b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" Jul 23 00:49:02.855470 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.855450 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344d7bc1-dd95-406d-a542-ef94377b648b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c\" (UID: \"344d7bc1-dd95-406d-a542-ef94377b648b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" Jul 23 00:49:02.956850 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.956811 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psrw7\" (UniqueName: \"kubernetes.io/projected/344d7bc1-dd95-406d-a542-ef94377b648b-kube-api-access-psrw7\") pod \"maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c\" (UID: \"344d7bc1-dd95-406d-a542-ef94377b648b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" Jul 23 00:49:02.956850 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.956857 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344d7bc1-dd95-406d-a542-ef94377b648b-maas-api-tls\") pod \"maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c\" (UID: \"344d7bc1-dd95-406d-a542-ef94377b648b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" Jul 23 00:49:02.957118 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.956888 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344d7bc1-dd95-406d-a542-ef94377b648b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c\" (UID: \"344d7bc1-dd95-406d-a542-ef94377b648b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" Jul 23 00:49:02.957118 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:49:02.956978 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-serving-cert: secret "maas-api-e2e-inf-a-e5bc11-serving-cert" not found Jul 23 00:49:02.957118 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:49:02.957052 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/344d7bc1-dd95-406d-a542-ef94377b648b-maas-api-tls podName:344d7bc1-dd95-406d-a542-ef94377b648b nodeName:}" failed. No retries permitted until 2026-07-23 00:49:03.457035386 +0000 UTC m=+2463.417773987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/344d7bc1-dd95-406d-a542-ef94377b648b-maas-api-tls") pod "maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" (UID: "344d7bc1-dd95-406d-a542-ef94377b648b") : secret "maas-api-e2e-inf-a-e5bc11-serving-cert" not found Jul 23 00:49:02.957614 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.957592 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344d7bc1-dd95-406d-a542-ef94377b648b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c\" (UID: \"344d7bc1-dd95-406d-a542-ef94377b648b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" Jul 23 00:49:02.974214 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:02.974178 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psrw7\" (UniqueName: \"kubernetes.io/projected/344d7bc1-dd95-406d-a542-ef94377b648b-kube-api-access-psrw7\") pod \"maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c\" (UID: \"344d7bc1-dd95-406d-a542-ef94377b648b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" Jul 23 00:49:03.462049 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:03.462013 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344d7bc1-dd95-406d-a542-ef94377b648b-maas-api-tls\") pod \"maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c\" (UID: \"344d7bc1-dd95-406d-a542-ef94377b648b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" Jul 23 00:49:03.464623 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:03.464591 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344d7bc1-dd95-406d-a542-ef94377b648b-maas-api-tls\") pod \"maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c\" (UID: \"344d7bc1-dd95-406d-a542-ef94377b648b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" Jul 23 00:49:03.651732 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:03.651695 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" Jul 23 00:49:03.672681 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:03.672652 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:03.788730 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:03.788686 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c"] Jul 23 00:49:03.791170 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:49:03.791145 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344d7bc1_dd95_406d_a542_ef94377b648b.slice/crio-657f2746af468db233d7fae14b0c765c623b92b9f13730f3a449be9ed31bd6f8 WatchSource:0}: Error finding container 657f2746af468db233d7fae14b0c765c623b92b9f13730f3a449be9ed31bd6f8: Status 404 returned error can't find the container with id 657f2746af468db233d7fae14b0c765c623b92b9f13730f3a449be9ed31bd6f8 Jul 23 00:49:04.577359 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:04.577320 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" event={"ID":"344d7bc1-dd95-406d-a542-ef94377b648b","Type":"ContainerStarted","Data":"2df0f08ddd84a2abe22c5f67e7e56611fef896a79e4d57197f991bd104c5bcf9"} Jul 23 00:49:04.577583 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:04.577381 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" event={"ID":"344d7bc1-dd95-406d-a542-ef94377b648b","Type":"ContainerStarted","Data":"657f2746af468db233d7fae14b0c765c623b92b9f13730f3a449be9ed31bd6f8"} Jul 23 00:49:04.577583 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:04.577435 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" Jul 23 00:49:04.594572 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:04.594517 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" podStartSLOduration=2.030911897 podStartE2EDuration="2.594501163s" totalImagesPullingTime="563.589266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:49:04.592320016 +0000 UTC m=+2464.553058636" watchObservedRunningTime="2026-07-23 00:49:04.594501163 +0000 UTC m=+2464.555239781" Jul 23 00:49:07.058411 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:07.058360 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:07.412563 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:07.412534 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:07.725228 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:07.725130 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:08.798082 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:08.798048 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:10.588917 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:10.588888 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" Jul 23 00:49:10.688274 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:10.688225 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:10.762835 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:10.762794 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:10.764004 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:10.763970 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv"] Jul 23 00:49:10.768045 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:10.768026 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" Jul 23 00:49:10.770220 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:10.770201 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a242b2-serving-cert\"" Jul 23 00:49:10.777220 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:10.777193 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv"] Jul 23 00:49:10.826155 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:10.826114 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ffcm\" (UniqueName: \"kubernetes.io/projected/5d5d0ccf-03cc-434e-846e-920428db5fae-kube-api-access-2ffcm\") pod \"maas-api-e2e-inf-b-a242b2-59d5869587-7zskv\" (UID: \"5d5d0ccf-03cc-434e-846e-920428db5fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" Jul 23 00:49:10.826155 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:10.826154 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d5d0ccf-03cc-434e-846e-920428db5fae-maas-api-tls\") pod \"maas-api-e2e-inf-b-a242b2-59d5869587-7zskv\" (UID: \"5d5d0ccf-03cc-434e-846e-920428db5fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" Jul 23 00:49:10.826488 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:10.826189 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5d0ccf-03cc-434e-846e-920428db5fae-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a242b2-59d5869587-7zskv\" (UID: \"5d5d0ccf-03cc-434e-846e-920428db5fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" Jul 23 00:49:10.827315 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:10.827292 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:10.926649 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:10.926623 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ffcm\" (UniqueName: \"kubernetes.io/projected/5d5d0ccf-03cc-434e-846e-920428db5fae-kube-api-access-2ffcm\") pod \"maas-api-e2e-inf-b-a242b2-59d5869587-7zskv\" (UID: \"5d5d0ccf-03cc-434e-846e-920428db5fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" Jul 23 00:49:10.926831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:10.926660 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d5d0ccf-03cc-434e-846e-920428db5fae-maas-api-tls\") pod \"maas-api-e2e-inf-b-a242b2-59d5869587-7zskv\" (UID: \"5d5d0ccf-03cc-434e-846e-920428db5fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" Jul 23 00:49:10.926831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:10.926694 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5d0ccf-03cc-434e-846e-920428db5fae-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a242b2-59d5869587-7zskv\" (UID: \"5d5d0ccf-03cc-434e-846e-920428db5fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" Jul 23 00:49:10.926831 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:49:10.926794 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-serving-cert: secret "maas-api-e2e-inf-b-a242b2-serving-cert" not found Jul 23 00:49:10.926996 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:49:10.926859 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d5d0ccf-03cc-434e-846e-920428db5fae-maas-api-tls podName:5d5d0ccf-03cc-434e-846e-920428db5fae nodeName:}" failed. No retries permitted until 2026-07-23 00:49:11.426837296 +0000 UTC m=+2471.387575915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d5d0ccf-03cc-434e-846e-920428db5fae-maas-api-tls") pod "maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" (UID: "5d5d0ccf-03cc-434e-846e-920428db5fae") : secret "maas-api-e2e-inf-b-a242b2-serving-cert" not found Jul 23 00:49:10.927341 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:10.927323 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5d0ccf-03cc-434e-846e-920428db5fae-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a242b2-59d5869587-7zskv\" (UID: \"5d5d0ccf-03cc-434e-846e-920428db5fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" Jul 23 00:49:10.935445 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:10.935422 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ffcm\" (UniqueName: \"kubernetes.io/projected/5d5d0ccf-03cc-434e-846e-920428db5fae-kube-api-access-2ffcm\") pod \"maas-api-e2e-inf-b-a242b2-59d5869587-7zskv\" (UID: \"5d5d0ccf-03cc-434e-846e-920428db5fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" Jul 23 00:49:11.212225 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:11.212139 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:11.311158 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:11.311112 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:11.409073 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:11.409039 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:11.432221 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:11.432181 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d5d0ccf-03cc-434e-846e-920428db5fae-maas-api-tls\") pod \"maas-api-e2e-inf-b-a242b2-59d5869587-7zskv\" (UID: \"5d5d0ccf-03cc-434e-846e-920428db5fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" Jul 23 00:49:11.434562 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:11.434546 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d5d0ccf-03cc-434e-846e-920428db5fae-maas-api-tls\") pod \"maas-api-e2e-inf-b-a242b2-59d5869587-7zskv\" (UID: \"5d5d0ccf-03cc-434e-846e-920428db5fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" Jul 23 00:49:11.679538 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:11.679504 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" Jul 23 00:49:11.811017 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:11.810983 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv"] Jul 23 00:49:11.814034 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:49:11.814006 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5d0ccf_03cc_434e_846e_920428db5fae.slice/crio-d014e978fa0caa69b05db1adbcf6fd12bd3065715eb368c7870044f79323ff3f WatchSource:0}: Error finding container d014e978fa0caa69b05db1adbcf6fd12bd3065715eb368c7870044f79323ff3f: Status 404 returned error can't find the container with id d014e978fa0caa69b05db1adbcf6fd12bd3065715eb368c7870044f79323ff3f Jul 23 00:49:12.617712 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:12.617677 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" event={"ID":"5d5d0ccf-03cc-434e-846e-920428db5fae","Type":"ContainerStarted","Data":"f26008e101e84bb379015f414677f23e6b16e5dce90ea53cc94bded908e6fab0"} Jul 23 00:49:12.617896 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:12.617726 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" event={"ID":"5d5d0ccf-03cc-434e-846e-920428db5fae","Type":"ContainerStarted","Data":"d014e978fa0caa69b05db1adbcf6fd12bd3065715eb368c7870044f79323ff3f"} Jul 23 00:49:12.617896 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:12.617816 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" Jul 23 00:49:12.632965 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:12.632913 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" podStartSLOduration=2.310120982 podStartE2EDuration="2.632894888s" totalImagesPullingTime="322.773906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:49:12.632166866 +0000 UTC m=+2472.592905509" watchObservedRunningTime="2026-07-23 00:49:12.632894888 +0000 UTC m=+2472.593633544" Jul 23 00:49:15.129963 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:15.129921 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:18.627869 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:18.627838 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" Jul 23 00:49:22.960710 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:22.960679 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:24.231508 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:24.231478 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:24.333859 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:24.333825 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:27.881414 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:27.881372 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:32.011221 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:32.011189 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:32.117114 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:32.117084 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:46.762065 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:49:46.762035 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:34.932556 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:34.932520 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:35.836901 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:35.836866 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:36.676059 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:36.676030 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:36.777583 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:36.777549 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:46.422784 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:46.422752 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c"] Jul 23 00:50:46.423248 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:46.422958 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" podUID="344d7bc1-dd95-406d-a542-ef94377b648b" containerName="maas-api" containerID="cri-o://2df0f08ddd84a2abe22c5f67e7e56611fef896a79e4d57197f991bd104c5bcf9" gracePeriod=30 Jul 23 00:50:46.461985 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:46.461955 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:46.865418 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:46.865378 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" Jul 23 00:50:46.996119 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:46.996038 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344d7bc1-dd95-406d-a542-ef94377b648b-maas-api-tls\") pod \"344d7bc1-dd95-406d-a542-ef94377b648b\" (UID: \"344d7bc1-dd95-406d-a542-ef94377b648b\") " Jul 23 00:50:46.996119 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:46.996091 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344d7bc1-dd95-406d-a542-ef94377b648b-openshift-service-ca\") pod \"344d7bc1-dd95-406d-a542-ef94377b648b\" (UID: \"344d7bc1-dd95-406d-a542-ef94377b648b\") " Jul 23 00:50:46.996307 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:46.996151 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psrw7\" (UniqueName: \"kubernetes.io/projected/344d7bc1-dd95-406d-a542-ef94377b648b-kube-api-access-psrw7\") pod \"344d7bc1-dd95-406d-a542-ef94377b648b\" (UID: \"344d7bc1-dd95-406d-a542-ef94377b648b\") " Jul 23 00:50:46.996532 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:46.996502 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/344d7bc1-dd95-406d-a542-ef94377b648b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "344d7bc1-dd95-406d-a542-ef94377b648b" (UID: "344d7bc1-dd95-406d-a542-ef94377b648b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:50:46.998122 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:46.998086 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/344d7bc1-dd95-406d-a542-ef94377b648b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "344d7bc1-dd95-406d-a542-ef94377b648b" (UID: "344d7bc1-dd95-406d-a542-ef94377b648b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:50:46.998214 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:46.998136 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344d7bc1-dd95-406d-a542-ef94377b648b-kube-api-access-psrw7" (OuterVolumeSpecName: "kube-api-access-psrw7") pod "344d7bc1-dd95-406d-a542-ef94377b648b" (UID: "344d7bc1-dd95-406d-a542-ef94377b648b"). InnerVolumeSpecName "kube-api-access-psrw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:50:47.061112 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:47.061080 2555 generic.go:356] "Generic (PLEG): container finished" podID="344d7bc1-dd95-406d-a542-ef94377b648b" containerID="2df0f08ddd84a2abe22c5f67e7e56611fef896a79e4d57197f991bd104c5bcf9" exitCode=0 Jul 23 00:50:47.061255 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:47.061127 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" Jul 23 00:50:47.061255 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:47.061162 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" event={"ID":"344d7bc1-dd95-406d-a542-ef94377b648b","Type":"ContainerDied","Data":"2df0f08ddd84a2abe22c5f67e7e56611fef896a79e4d57197f991bd104c5bcf9"} Jul 23 00:50:47.061255 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:47.061220 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c" event={"ID":"344d7bc1-dd95-406d-a542-ef94377b648b","Type":"ContainerDied","Data":"657f2746af468db233d7fae14b0c765c623b92b9f13730f3a449be9ed31bd6f8"} Jul 23 00:50:47.061255 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:47.061249 2555 scope.go:117] "RemoveContainer" containerID="2df0f08ddd84a2abe22c5f67e7e56611fef896a79e4d57197f991bd104c5bcf9" Jul 23 00:50:47.069888 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:47.069873 2555 scope.go:117] "RemoveContainer" containerID="2df0f08ddd84a2abe22c5f67e7e56611fef896a79e4d57197f991bd104c5bcf9" Jul 23 00:50:47.070141 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:50:47.070121 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df0f08ddd84a2abe22c5f67e7e56611fef896a79e4d57197f991bd104c5bcf9\": container with ID starting with 2df0f08ddd84a2abe22c5f67e7e56611fef896a79e4d57197f991bd104c5bcf9 not found: ID does not exist" containerID="2df0f08ddd84a2abe22c5f67e7e56611fef896a79e4d57197f991bd104c5bcf9" Jul 23 00:50:47.070216 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:47.070153 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df0f08ddd84a2abe22c5f67e7e56611fef896a79e4d57197f991bd104c5bcf9"} err="failed to get container status \"2df0f08ddd84a2abe22c5f67e7e56611fef896a79e4d57197f991bd104c5bcf9\": rpc error: code = NotFound desc = could not find container \"2df0f08ddd84a2abe22c5f67e7e56611fef896a79e4d57197f991bd104c5bcf9\": container with ID starting with 2df0f08ddd84a2abe22c5f67e7e56611fef896a79e4d57197f991bd104c5bcf9 not found: ID does not exist" Jul 23 00:50:47.084166 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:47.084137 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c"] Jul 23 00:50:47.086475 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:47.086452 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5bc11-54894bbfd6-cbq8c"] Jul 23 00:50:47.097921 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:47.097883 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344d7bc1-dd95-406d-a542-ef94377b648b-maas-api-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:50:47.097921 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:47.097918 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344d7bc1-dd95-406d-a542-ef94377b648b-openshift-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:50:47.098099 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:47.097931 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psrw7\" (UniqueName: \"kubernetes.io/projected/344d7bc1-dd95-406d-a542-ef94377b648b-kube-api-access-psrw7\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:50:48.502321 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:48.502291 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="344d7bc1-dd95-406d-a542-ef94377b648b" path="/var/lib/kubelet/pods/344d7bc1-dd95-406d-a542-ef94377b648b/volumes" Jul 23 00:50:51.084100 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:51.084070 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:51.507221 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:51.507192 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:57.532022 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:57.531994 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:57.606944 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:57.606914 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:57.895841 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:57.895757 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:58.060990 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:58.060960 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:58.914657 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:58.914621 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:59.013794 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:50:59.013768 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:08.623378 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:08.623339 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv"] Jul 23 00:51:08.623957 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:08.623613 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" podUID="5d5d0ccf-03cc-434e-846e-920428db5fae" containerName="maas-api" containerID="cri-o://f26008e101e84bb379015f414677f23e6b16e5dce90ea53cc94bded908e6fab0" gracePeriod=30 Jul 23 00:51:08.650984 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:08.650954 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:09.075280 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.075257 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" Jul 23 00:51:09.164818 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.164783 2555 generic.go:356] "Generic (PLEG): container finished" podID="5d5d0ccf-03cc-434e-846e-920428db5fae" containerID="f26008e101e84bb379015f414677f23e6b16e5dce90ea53cc94bded908e6fab0" exitCode=0 Jul 23 00:51:09.165013 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.164827 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" Jul 23 00:51:09.165013 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.164870 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" event={"ID":"5d5d0ccf-03cc-434e-846e-920428db5fae","Type":"ContainerDied","Data":"f26008e101e84bb379015f414677f23e6b16e5dce90ea53cc94bded908e6fab0"} Jul 23 00:51:09.165013 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.164926 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv" event={"ID":"5d5d0ccf-03cc-434e-846e-920428db5fae","Type":"ContainerDied","Data":"d014e978fa0caa69b05db1adbcf6fd12bd3065715eb368c7870044f79323ff3f"} Jul 23 00:51:09.165013 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.164949 2555 scope.go:117] "RemoveContainer" containerID="f26008e101e84bb379015f414677f23e6b16e5dce90ea53cc94bded908e6fab0" Jul 23 00:51:09.176182 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.176158 2555 scope.go:117] "RemoveContainer" containerID="f26008e101e84bb379015f414677f23e6b16e5dce90ea53cc94bded908e6fab0" Jul 23 00:51:09.176482 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:51:09.176465 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26008e101e84bb379015f414677f23e6b16e5dce90ea53cc94bded908e6fab0\": container with ID starting with f26008e101e84bb379015f414677f23e6b16e5dce90ea53cc94bded908e6fab0 not found: ID does not exist" containerID="f26008e101e84bb379015f414677f23e6b16e5dce90ea53cc94bded908e6fab0" Jul 23 00:51:09.176537 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.176491 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26008e101e84bb379015f414677f23e6b16e5dce90ea53cc94bded908e6fab0"} err="failed to get container status \"f26008e101e84bb379015f414677f23e6b16e5dce90ea53cc94bded908e6fab0\": rpc error: code = NotFound desc = could not find container \"f26008e101e84bb379015f414677f23e6b16e5dce90ea53cc94bded908e6fab0\": container with ID starting with f26008e101e84bb379015f414677f23e6b16e5dce90ea53cc94bded908e6fab0 not found: ID does not exist" Jul 23 00:51:09.177718 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.177701 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ffcm\" (UniqueName: \"kubernetes.io/projected/5d5d0ccf-03cc-434e-846e-920428db5fae-kube-api-access-2ffcm\") pod \"5d5d0ccf-03cc-434e-846e-920428db5fae\" (UID: \"5d5d0ccf-03cc-434e-846e-920428db5fae\") " Jul 23 00:51:09.177770 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.177738 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d5d0ccf-03cc-434e-846e-920428db5fae-maas-api-tls\") pod \"5d5d0ccf-03cc-434e-846e-920428db5fae\" (UID: \"5d5d0ccf-03cc-434e-846e-920428db5fae\") " Jul 23 00:51:09.177770 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.177756 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5d0ccf-03cc-434e-846e-920428db5fae-openshift-service-ca\") pod \"5d5d0ccf-03cc-434e-846e-920428db5fae\" (UID: \"5d5d0ccf-03cc-434e-846e-920428db5fae\") " Jul 23 00:51:09.178117 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.178096 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5d0ccf-03cc-434e-846e-920428db5fae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5d5d0ccf-03cc-434e-846e-920428db5fae" (UID: "5d5d0ccf-03cc-434e-846e-920428db5fae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:51:09.179831 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.179810 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d5d0ccf-03cc-434e-846e-920428db5fae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5d5d0ccf-03cc-434e-846e-920428db5fae" (UID: "5d5d0ccf-03cc-434e-846e-920428db5fae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:51:09.179913 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.179854 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5d0ccf-03cc-434e-846e-920428db5fae-kube-api-access-2ffcm" (OuterVolumeSpecName: "kube-api-access-2ffcm") pod "5d5d0ccf-03cc-434e-846e-920428db5fae" (UID: "5d5d0ccf-03cc-434e-846e-920428db5fae"). InnerVolumeSpecName "kube-api-access-2ffcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:51:09.279431 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.279379 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ffcm\" (UniqueName: \"kubernetes.io/projected/5d5d0ccf-03cc-434e-846e-920428db5fae-kube-api-access-2ffcm\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:51:09.279431 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.279425 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d5d0ccf-03cc-434e-846e-920428db5fae-maas-api-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:51:09.279431 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.279434 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5d0ccf-03cc-434e-846e-920428db5fae-openshift-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:51:09.486122 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.486082 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv"] Jul 23 00:51:09.490361 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:09.490325 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a242b2-59d5869587-7zskv"] Jul 23 00:51:10.502459 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:10.502427 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d5d0ccf-03cc-434e-846e-920428db5fae" path="/var/lib/kubelet/pods/5d5d0ccf-03cc-434e-846e-920428db5fae/volumes" Jul 23 00:51:13.257409 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:13.257348 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:13.677038 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:13.677006 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:19.481506 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:19.481473 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:19.540898 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:19.540847 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:19.838679 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:19.838590 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:20.039738 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:20.039708 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:20.386125 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:20.386094 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:20.735000 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:20.734953 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:21.326926 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:21.326896 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:23.397439 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.397413 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:23.488053 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.488019 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5"] Jul 23 00:51:23.488579 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.488563 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d5d0ccf-03cc-434e-846e-920428db5fae" containerName="maas-api" Jul 23 00:51:23.488660 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.488582 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d5d0ccf-03cc-434e-846e-920428db5fae" containerName="maas-api" Jul 23 00:51:23.488660 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.488600 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="344d7bc1-dd95-406d-a542-ef94377b648b" containerName="maas-api" Jul 23 00:51:23.488660 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.488608 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="344d7bc1-dd95-406d-a542-ef94377b648b" containerName="maas-api" Jul 23 00:51:23.488827 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.488693 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d5d0ccf-03cc-434e-846e-920428db5fae" containerName="maas-api" Jul 23 00:51:23.488827 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.488705 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="344d7bc1-dd95-406d-a542-ef94377b648b" containerName="maas-api" Jul 23 00:51:23.493402 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.493371 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" Jul 23 00:51:23.495728 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.495707 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gktdb\"" Jul 23 00:51:23.496045 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.496026 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-16466f-serving-cert\"" Jul 23 00:51:23.498498 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.498424 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:23.500276 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.500254 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5"] Jul 23 00:51:23.591470 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.591421 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:23.614946 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.614918 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5484a0df-40b8-418b-a2ae-4e7b8503d40d-openshift-service-ca\") pod \"maas-api-e2e-api-a-16466f-d94f455d5-ch6p5\" (UID: \"5484a0df-40b8-418b-a2ae-4e7b8503d40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" Jul 23 00:51:23.615123 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.614969 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zmht\" (UniqueName: \"kubernetes.io/projected/5484a0df-40b8-418b-a2ae-4e7b8503d40d-kube-api-access-6zmht\") pod \"maas-api-e2e-api-a-16466f-d94f455d5-ch6p5\" (UID: \"5484a0df-40b8-418b-a2ae-4e7b8503d40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" Jul 23 00:51:23.615123 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.615088 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5484a0df-40b8-418b-a2ae-4e7b8503d40d-maas-api-tls\") pod \"maas-api-e2e-api-a-16466f-d94f455d5-ch6p5\" (UID: \"5484a0df-40b8-418b-a2ae-4e7b8503d40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" Jul 23 00:51:23.716514 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.716485 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5484a0df-40b8-418b-a2ae-4e7b8503d40d-maas-api-tls\") pod \"maas-api-e2e-api-a-16466f-d94f455d5-ch6p5\" (UID: \"5484a0df-40b8-418b-a2ae-4e7b8503d40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" Jul 23 00:51:23.716701 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.716545 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5484a0df-40b8-418b-a2ae-4e7b8503d40d-openshift-service-ca\") pod \"maas-api-e2e-api-a-16466f-d94f455d5-ch6p5\" (UID: \"5484a0df-40b8-418b-a2ae-4e7b8503d40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" Jul 23 00:51:23.716701 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:51:23.716650 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-serving-cert: secret "maas-api-e2e-api-a-16466f-serving-cert" not found Jul 23 00:51:23.716807 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.716653 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zmht\" (UniqueName: \"kubernetes.io/projected/5484a0df-40b8-418b-a2ae-4e7b8503d40d-kube-api-access-6zmht\") pod \"maas-api-e2e-api-a-16466f-d94f455d5-ch6p5\" (UID: \"5484a0df-40b8-418b-a2ae-4e7b8503d40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" Jul 23 00:51:23.716807 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:51:23.716730 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5484a0df-40b8-418b-a2ae-4e7b8503d40d-maas-api-tls podName:5484a0df-40b8-418b-a2ae-4e7b8503d40d nodeName:}" failed. No retries permitted until 2026-07-23 00:51:24.216711033 +0000 UTC m=+2604.177449630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5484a0df-40b8-418b-a2ae-4e7b8503d40d-maas-api-tls") pod "maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" (UID: "5484a0df-40b8-418b-a2ae-4e7b8503d40d") : secret "maas-api-e2e-api-a-16466f-serving-cert" not found Jul 23 00:51:23.717123 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.717106 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5484a0df-40b8-418b-a2ae-4e7b8503d40d-openshift-service-ca\") pod \"maas-api-e2e-api-a-16466f-d94f455d5-ch6p5\" (UID: \"5484a0df-40b8-418b-a2ae-4e7b8503d40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" Jul 23 00:51:23.724863 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:23.724838 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zmht\" (UniqueName: \"kubernetes.io/projected/5484a0df-40b8-418b-a2ae-4e7b8503d40d-kube-api-access-6zmht\") pod \"maas-api-e2e-api-a-16466f-d94f455d5-ch6p5\" (UID: \"5484a0df-40b8-418b-a2ae-4e7b8503d40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" Jul 23 00:51:24.013178 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:24.013084 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:24.118104 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:24.118069 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:24.213786 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:24.213745 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:24.220975 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:24.220949 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5484a0df-40b8-418b-a2ae-4e7b8503d40d-maas-api-tls\") pod \"maas-api-e2e-api-a-16466f-d94f455d5-ch6p5\" (UID: \"5484a0df-40b8-418b-a2ae-4e7b8503d40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" Jul 23 00:51:24.223488 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:24.223470 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5484a0df-40b8-418b-a2ae-4e7b8503d40d-maas-api-tls\") pod \"maas-api-e2e-api-a-16466f-d94f455d5-ch6p5\" (UID: \"5484a0df-40b8-418b-a2ae-4e7b8503d40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" Jul 23 00:51:24.405774 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:24.405688 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" Jul 23 00:51:24.542916 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:24.542890 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5"] Jul 23 00:51:24.544591 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:51:24.544561 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5484a0df_40b8_418b_a2ae_4e7b8503d40d.slice/crio-d3d955b343a46d4bcc60078468f8d8808ca3d76aee8e41c9cb5f77409f9c621b WatchSource:0}: Error finding container d3d955b343a46d4bcc60078468f8d8808ca3d76aee8e41c9cb5f77409f9c621b: Status 404 returned error can't find the container with id d3d955b343a46d4bcc60078468f8d8808ca3d76aee8e41c9cb5f77409f9c621b Jul 23 00:51:24.545829 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:24.545813 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:51:25.239738 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:25.239701 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" event={"ID":"5484a0df-40b8-418b-a2ae-4e7b8503d40d","Type":"ContainerStarted","Data":"b9cbe7bbc5dfda077dbf8d9ce6c396ffd2dcb0298cacc9ef71561c3f1010d83a"} Jul 23 00:51:25.239965 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:25.239757 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" Jul 23 00:51:25.239965 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:25.239776 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" event={"ID":"5484a0df-40b8-418b-a2ae-4e7b8503d40d","Type":"ContainerStarted","Data":"d3d955b343a46d4bcc60078468f8d8808ca3d76aee8e41c9cb5f77409f9c621b"} Jul 23 00:51:25.265073 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:25.265024 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" podStartSLOduration=1.7872558010000001 podStartE2EDuration="2.265008845s" totalImagesPullingTime="477.753044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:51:25.263058286 +0000 UTC m=+2605.223796897" watchObservedRunningTime="2026-07-23 00:51:25.265008845 +0000 UTC m=+2605.225747463" Jul 23 00:51:27.927630 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:27.927569 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:28.416609 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:28.416579 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:28.756344 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:28.756261 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:29.042706 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:29.042617 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:31.251076 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.251045 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" Jul 23 00:51:31.453374 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.453333 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:31.545456 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.536358 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x"] Jul 23 00:51:31.545456 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.541125 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" Jul 23 00:51:31.545456 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.543756 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e2debd-serving-cert\"" Jul 23 00:51:31.547141 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.547112 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:31.549545 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.549514 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x"] Jul 23 00:51:31.622294 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.622253 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:31.688790 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.688745 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkhgw\" (UniqueName: \"kubernetes.io/projected/ffb14dfa-875f-491e-ad46-ccd3e085b221-kube-api-access-rkhgw\") pod \"maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x\" (UID: \"ffb14dfa-875f-491e-ad46-ccd3e085b221\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" Jul 23 00:51:31.688790 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.688795 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffb14dfa-875f-491e-ad46-ccd3e085b221-maas-api-tls\") pod \"maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x\" (UID: \"ffb14dfa-875f-491e-ad46-ccd3e085b221\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" Jul 23 00:51:31.688997 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.688874 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb14dfa-875f-491e-ad46-ccd3e085b221-openshift-service-ca\") pod \"maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x\" (UID: \"ffb14dfa-875f-491e-ad46-ccd3e085b221\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" Jul 23 00:51:31.719598 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.719563 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:31.789992 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.789949 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkhgw\" (UniqueName: \"kubernetes.io/projected/ffb14dfa-875f-491e-ad46-ccd3e085b221-kube-api-access-rkhgw\") pod \"maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x\" (UID: \"ffb14dfa-875f-491e-ad46-ccd3e085b221\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" Jul 23 00:51:31.789992 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.789993 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffb14dfa-875f-491e-ad46-ccd3e085b221-maas-api-tls\") pod \"maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x\" (UID: \"ffb14dfa-875f-491e-ad46-ccd3e085b221\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" Jul 23 00:51:31.790225 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.790024 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb14dfa-875f-491e-ad46-ccd3e085b221-openshift-service-ca\") pod \"maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x\" (UID: \"ffb14dfa-875f-491e-ad46-ccd3e085b221\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" Jul 23 00:51:31.790815 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.790792 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb14dfa-875f-491e-ad46-ccd3e085b221-openshift-service-ca\") pod \"maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x\" (UID: \"ffb14dfa-875f-491e-ad46-ccd3e085b221\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" Jul 23 00:51:31.792492 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.792468 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffb14dfa-875f-491e-ad46-ccd3e085b221-maas-api-tls\") pod \"maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x\" (UID: \"ffb14dfa-875f-491e-ad46-ccd3e085b221\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" Jul 23 00:51:31.797998 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.797946 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkhgw\" (UniqueName: \"kubernetes.io/projected/ffb14dfa-875f-491e-ad46-ccd3e085b221-kube-api-access-rkhgw\") pod \"maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x\" (UID: \"ffb14dfa-875f-491e-ad46-ccd3e085b221\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" Jul 23 00:51:31.864345 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:31.864307 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" Jul 23 00:51:32.002922 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:32.002892 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x"] Jul 23 00:51:32.004905 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:51:32.004882 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb14dfa_875f_491e_ad46_ccd3e085b221.slice/crio-87957d97c1bd7ef56a912a8178ac545bc1cc8d24ac47dc12608237361865cb8a WatchSource:0}: Error finding container 87957d97c1bd7ef56a912a8178ac545bc1cc8d24ac47dc12608237361865cb8a: Status 404 returned error can't find the container with id 87957d97c1bd7ef56a912a8178ac545bc1cc8d24ac47dc12608237361865cb8a Jul 23 00:51:32.217236 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:32.217208 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:32.272539 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:32.272507 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" event={"ID":"ffb14dfa-875f-491e-ad46-ccd3e085b221","Type":"ContainerStarted","Data":"87957d97c1bd7ef56a912a8178ac545bc1cc8d24ac47dc12608237361865cb8a"} Jul 23 00:51:32.305660 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:32.305629 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:33.279130 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:33.279091 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" event={"ID":"ffb14dfa-875f-491e-ad46-ccd3e085b221","Type":"ContainerStarted","Data":"620c815b862e2c7b7d014c4adddeceaf618740edda2cab38038160c4b2198e7f"} Jul 23 00:51:33.279564 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:33.279163 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" Jul 23 00:51:33.298272 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:33.298222 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" podStartSLOduration=1.9254381280000001 podStartE2EDuration="2.298207597s" totalImagesPullingTime="372.769469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:51:33.296150013 +0000 UTC m=+2613.256888632" watchObservedRunningTime="2026-07-23 00:51:33.298207597 +0000 UTC m=+2613.258946215" Jul 23 00:51:35.955006 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:35.954969 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:39.289651 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:39.289624 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" Jul 23 00:51:44.745026 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:44.744987 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:44.856346 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:44.856312 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:53.271078 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:53.271041 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:53.369685 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:53.369648 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:54.671946 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:54.671905 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:54.774739 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:54.774699 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:59.915656 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:59.915615 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5"] Jul 23 00:51:59.916230 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:59.915816 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" podUID="5484a0df-40b8-418b-a2ae-4e7b8503d40d" containerName="maas-api" containerID="cri-o://b9cbe7bbc5dfda077dbf8d9ce6c396ffd2dcb0298cacc9ef71561c3f1010d83a" gracePeriod=30 Jul 23 00:51:59.942077 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:51:59.942044 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:00.160840 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.160819 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" Jul 23 00:52:00.251497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.251382 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zmht\" (UniqueName: \"kubernetes.io/projected/5484a0df-40b8-418b-a2ae-4e7b8503d40d-kube-api-access-6zmht\") pod \"5484a0df-40b8-418b-a2ae-4e7b8503d40d\" (UID: \"5484a0df-40b8-418b-a2ae-4e7b8503d40d\") " Jul 23 00:52:00.251497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.251461 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5484a0df-40b8-418b-a2ae-4e7b8503d40d-openshift-service-ca\") pod \"5484a0df-40b8-418b-a2ae-4e7b8503d40d\" (UID: \"5484a0df-40b8-418b-a2ae-4e7b8503d40d\") " Jul 23 00:52:00.251497 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.251500 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5484a0df-40b8-418b-a2ae-4e7b8503d40d-maas-api-tls\") pod \"5484a0df-40b8-418b-a2ae-4e7b8503d40d\" (UID: \"5484a0df-40b8-418b-a2ae-4e7b8503d40d\") " Jul 23 00:52:00.251907 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.251878 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5484a0df-40b8-418b-a2ae-4e7b8503d40d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5484a0df-40b8-418b-a2ae-4e7b8503d40d" (UID: "5484a0df-40b8-418b-a2ae-4e7b8503d40d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:52:00.253546 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.253521 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5484a0df-40b8-418b-a2ae-4e7b8503d40d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5484a0df-40b8-418b-a2ae-4e7b8503d40d" (UID: "5484a0df-40b8-418b-a2ae-4e7b8503d40d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:52:00.253546 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.253535 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5484a0df-40b8-418b-a2ae-4e7b8503d40d-kube-api-access-6zmht" (OuterVolumeSpecName: "kube-api-access-6zmht") pod "5484a0df-40b8-418b-a2ae-4e7b8503d40d" (UID: "5484a0df-40b8-418b-a2ae-4e7b8503d40d"). InnerVolumeSpecName "kube-api-access-6zmht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:52:00.352574 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.352540 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zmht\" (UniqueName: \"kubernetes.io/projected/5484a0df-40b8-418b-a2ae-4e7b8503d40d-kube-api-access-6zmht\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:52:00.352574 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.352566 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5484a0df-40b8-418b-a2ae-4e7b8503d40d-openshift-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:52:00.352574 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.352576 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5484a0df-40b8-418b-a2ae-4e7b8503d40d-maas-api-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:52:00.407885 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.407848 2555 generic.go:356] "Generic (PLEG): container finished" podID="5484a0df-40b8-418b-a2ae-4e7b8503d40d" containerID="b9cbe7bbc5dfda077dbf8d9ce6c396ffd2dcb0298cacc9ef71561c3f1010d83a" exitCode=0 Jul 23 00:52:00.408086 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.407901 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" event={"ID":"5484a0df-40b8-418b-a2ae-4e7b8503d40d","Type":"ContainerDied","Data":"b9cbe7bbc5dfda077dbf8d9ce6c396ffd2dcb0298cacc9ef71561c3f1010d83a"} Jul 23 00:52:00.408086 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.407914 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" Jul 23 00:52:00.408086 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.407966 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5" event={"ID":"5484a0df-40b8-418b-a2ae-4e7b8503d40d","Type":"ContainerDied","Data":"d3d955b343a46d4bcc60078468f8d8808ca3d76aee8e41c9cb5f77409f9c621b"} Jul 23 00:52:00.408086 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.407989 2555 scope.go:117] "RemoveContainer" containerID="b9cbe7bbc5dfda077dbf8d9ce6c396ffd2dcb0298cacc9ef71561c3f1010d83a" Jul 23 00:52:00.417409 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.417372 2555 scope.go:117] "RemoveContainer" containerID="b9cbe7bbc5dfda077dbf8d9ce6c396ffd2dcb0298cacc9ef71561c3f1010d83a" Jul 23 00:52:00.417657 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:52:00.417638 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9cbe7bbc5dfda077dbf8d9ce6c396ffd2dcb0298cacc9ef71561c3f1010d83a\": container with ID starting with b9cbe7bbc5dfda077dbf8d9ce6c396ffd2dcb0298cacc9ef71561c3f1010d83a not found: ID does not exist" containerID="b9cbe7bbc5dfda077dbf8d9ce6c396ffd2dcb0298cacc9ef71561c3f1010d83a" Jul 23 00:52:00.417719 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.417665 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9cbe7bbc5dfda077dbf8d9ce6c396ffd2dcb0298cacc9ef71561c3f1010d83a"} err="failed to get container status \"b9cbe7bbc5dfda077dbf8d9ce6c396ffd2dcb0298cacc9ef71561c3f1010d83a\": rpc error: code = NotFound desc = could not find container \"b9cbe7bbc5dfda077dbf8d9ce6c396ffd2dcb0298cacc9ef71561c3f1010d83a\": container with ID starting with b9cbe7bbc5dfda077dbf8d9ce6c396ffd2dcb0298cacc9ef71561c3f1010d83a not found: ID does not exist" Jul 23 00:52:00.430490 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.430453 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5"] Jul 23 00:52:00.436809 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.436783 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-16466f-d94f455d5-ch6p5"] Jul 23 00:52:00.504044 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:00.503971 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5484a0df-40b8-418b-a2ae-4e7b8503d40d" path="/var/lib/kubelet/pods/5484a0df-40b8-418b-a2ae-4e7b8503d40d/volumes" Jul 23 00:52:04.215822 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:04.215790 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:04.608971 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:04.608895 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:10.709131 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:10.709098 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:10.778705 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:10.778674 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:10.983507 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:10.983413 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:11.208853 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:11.208823 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:11.858232 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:11.858202 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:11.955966 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:11.955919 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:16.175584 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:16.175553 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:16.246016 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:16.245974 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:21.868609 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:21.868561 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:21.970322 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:21.970279 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:25.869590 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:25.869553 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:25.971820 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:25.971790 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:27.108474 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.108437 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x"] Jul 23 00:52:27.108905 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.108680 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" podUID="ffb14dfa-875f-491e-ad46-ccd3e085b221" containerName="maas-api" containerID="cri-o://620c815b862e2c7b7d014c4adddeceaf618740edda2cab38038160c4b2198e7f" gracePeriod=30 Jul 23 00:52:27.129251 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.129224 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:27.360103 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.360041 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" Jul 23 00:52:27.404346 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.404319 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkhgw\" (UniqueName: \"kubernetes.io/projected/ffb14dfa-875f-491e-ad46-ccd3e085b221-kube-api-access-rkhgw\") pod \"ffb14dfa-875f-491e-ad46-ccd3e085b221\" (UID: \"ffb14dfa-875f-491e-ad46-ccd3e085b221\") " Jul 23 00:52:27.404553 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.404381 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffb14dfa-875f-491e-ad46-ccd3e085b221-maas-api-tls\") pod \"ffb14dfa-875f-491e-ad46-ccd3e085b221\" (UID: \"ffb14dfa-875f-491e-ad46-ccd3e085b221\") " Jul 23 00:52:27.404553 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.404472 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb14dfa-875f-491e-ad46-ccd3e085b221-openshift-service-ca\") pod \"ffb14dfa-875f-491e-ad46-ccd3e085b221\" (UID: \"ffb14dfa-875f-491e-ad46-ccd3e085b221\") " Jul 23 00:52:27.404913 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.404887 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb14dfa-875f-491e-ad46-ccd3e085b221-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ffb14dfa-875f-491e-ad46-ccd3e085b221" (UID: "ffb14dfa-875f-491e-ad46-ccd3e085b221"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:52:27.406462 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.406439 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb14dfa-875f-491e-ad46-ccd3e085b221-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ffb14dfa-875f-491e-ad46-ccd3e085b221" (UID: "ffb14dfa-875f-491e-ad46-ccd3e085b221"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:52:27.406566 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.406489 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb14dfa-875f-491e-ad46-ccd3e085b221-kube-api-access-rkhgw" (OuterVolumeSpecName: "kube-api-access-rkhgw") pod "ffb14dfa-875f-491e-ad46-ccd3e085b221" (UID: "ffb14dfa-875f-491e-ad46-ccd3e085b221"). InnerVolumeSpecName "kube-api-access-rkhgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:52:27.505760 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.505729 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb14dfa-875f-491e-ad46-ccd3e085b221-openshift-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:52:27.505760 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.505756 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkhgw\" (UniqueName: \"kubernetes.io/projected/ffb14dfa-875f-491e-ad46-ccd3e085b221-kube-api-access-rkhgw\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:52:27.505760 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.505766 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffb14dfa-875f-491e-ad46-ccd3e085b221-maas-api-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:52:27.533744 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.533716 2555 generic.go:356] "Generic (PLEG): container finished" podID="ffb14dfa-875f-491e-ad46-ccd3e085b221" containerID="620c815b862e2c7b7d014c4adddeceaf618740edda2cab38038160c4b2198e7f" exitCode=0 Jul 23 00:52:27.533882 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.533762 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" Jul 23 00:52:27.533882 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.533807 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" event={"ID":"ffb14dfa-875f-491e-ad46-ccd3e085b221","Type":"ContainerDied","Data":"620c815b862e2c7b7d014c4adddeceaf618740edda2cab38038160c4b2198e7f"} Jul 23 00:52:27.533882 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.533861 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x" event={"ID":"ffb14dfa-875f-491e-ad46-ccd3e085b221","Type":"ContainerDied","Data":"87957d97c1bd7ef56a912a8178ac545bc1cc8d24ac47dc12608237361865cb8a"} Jul 23 00:52:27.533997 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.533890 2555 scope.go:117] "RemoveContainer" containerID="620c815b862e2c7b7d014c4adddeceaf618740edda2cab38038160c4b2198e7f" Jul 23 00:52:27.544643 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.544628 2555 scope.go:117] "RemoveContainer" containerID="620c815b862e2c7b7d014c4adddeceaf618740edda2cab38038160c4b2198e7f" Jul 23 00:52:27.545109 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:52:27.545090 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"620c815b862e2c7b7d014c4adddeceaf618740edda2cab38038160c4b2198e7f\": container with ID starting with 620c815b862e2c7b7d014c4adddeceaf618740edda2cab38038160c4b2198e7f not found: ID does not exist" containerID="620c815b862e2c7b7d014c4adddeceaf618740edda2cab38038160c4b2198e7f" Jul 23 00:52:27.545203 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.545116 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"620c815b862e2c7b7d014c4adddeceaf618740edda2cab38038160c4b2198e7f"} err="failed to get container status \"620c815b862e2c7b7d014c4adddeceaf618740edda2cab38038160c4b2198e7f\": rpc error: code = NotFound desc = could not find container \"620c815b862e2c7b7d014c4adddeceaf618740edda2cab38038160c4b2198e7f\": container with ID starting with 620c815b862e2c7b7d014c4adddeceaf618740edda2cab38038160c4b2198e7f not found: ID does not exist" Jul 23 00:52:27.554557 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.554535 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x"] Jul 23 00:52:27.559051 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:27.559033 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e2debd-5869c5cbfc-qfq4x"] Jul 23 00:52:28.503441 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:28.503410 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffb14dfa-875f-491e-ad46-ccd3e085b221" path="/var/lib/kubelet/pods/ffb14dfa-875f-491e-ad46-ccd3e085b221/volumes" Jul 23 00:52:31.407641 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:31.407612 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:31.821625 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:31.821593 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:37.614440 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:37.614383 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:37.717548 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:37.717510 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:37.902819 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:37.902741 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:38.203505 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:38.203463 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:38.526181 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:38.526088 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:38.946888 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:38.946856 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:39.123036 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:39.122999 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:41.759922 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:41.759882 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:41.856184 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:41.856143 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:41.862431 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:41.862403 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72"] Jul 23 00:52:41.862814 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:41.862800 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffb14dfa-875f-491e-ad46-ccd3e085b221" containerName="maas-api" Jul 23 00:52:41.862878 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:41.862815 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffb14dfa-875f-491e-ad46-ccd3e085b221" containerName="maas-api" Jul 23 00:52:41.862878 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:41.862830 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5484a0df-40b8-418b-a2ae-4e7b8503d40d" containerName="maas-api" Jul 23 00:52:41.862878 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:41.862835 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5484a0df-40b8-418b-a2ae-4e7b8503d40d" containerName="maas-api" Jul 23 00:52:41.863031 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:41.862902 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5484a0df-40b8-418b-a2ae-4e7b8503d40d" containerName="maas-api" Jul 23 00:52:41.863031 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:41.862912 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffb14dfa-875f-491e-ad46-ccd3e085b221" containerName="maas-api" Jul 23 00:52:41.865953 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:41.865935 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" Jul 23 00:52:41.868466 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:41.868450 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gktdb\"" Jul 23 00:52:41.868615 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:41.868599 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-c75140-serving-cert\"" Jul 23 00:52:41.878278 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:41.878254 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72"] Jul 23 00:52:41.939362 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:41.939327 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5743a3c-5a25-4705-807b-b55376037dc2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72\" (UID: \"e5743a3c-5a25-4705-807b-b55376037dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" Jul 23 00:52:41.939591 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:41.939386 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5743a3c-5a25-4705-807b-b55376037dc2-maas-api-tls\") pod \"maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72\" (UID: \"e5743a3c-5a25-4705-807b-b55376037dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" Jul 23 00:52:41.939591 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:41.939561 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sdcr\" (UniqueName: \"kubernetes.io/projected/e5743a3c-5a25-4705-807b-b55376037dc2-kube-api-access-2sdcr\") pod \"maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72\" (UID: \"e5743a3c-5a25-4705-807b-b55376037dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" Jul 23 00:52:42.040240 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:42.040145 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5743a3c-5a25-4705-807b-b55376037dc2-maas-api-tls\") pod \"maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72\" (UID: \"e5743a3c-5a25-4705-807b-b55376037dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" Jul 23 00:52:42.040240 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:42.040211 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sdcr\" (UniqueName: \"kubernetes.io/projected/e5743a3c-5a25-4705-807b-b55376037dc2-kube-api-access-2sdcr\") pod \"maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72\" (UID: \"e5743a3c-5a25-4705-807b-b55376037dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" Jul 23 00:52:42.040513 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:42.040290 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5743a3c-5a25-4705-807b-b55376037dc2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72\" (UID: \"e5743a3c-5a25-4705-807b-b55376037dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" Jul 23 00:52:42.040854 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:42.040825 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:42.040991 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:42.040972 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5743a3c-5a25-4705-807b-b55376037dc2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72\" (UID: \"e5743a3c-5a25-4705-807b-b55376037dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" Jul 23 00:52:42.042689 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:42.042674 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5743a3c-5a25-4705-807b-b55376037dc2-maas-api-tls\") pod \"maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72\" (UID: \"e5743a3c-5a25-4705-807b-b55376037dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" Jul 23 00:52:42.055213 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:42.055188 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sdcr\" (UniqueName: \"kubernetes.io/projected/e5743a3c-5a25-4705-807b-b55376037dc2-kube-api-access-2sdcr\") pod \"maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72\" (UID: \"e5743a3c-5a25-4705-807b-b55376037dc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" Jul 23 00:52:42.149700 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:42.149670 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:42.176759 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:42.176727 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" Jul 23 00:52:42.315582 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:42.315558 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72"] Jul 23 00:52:42.317601 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:52:42.317567 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5743a3c_5a25_4705_807b_b55376037dc2.slice/crio-d86e67088ad3833f163a70addc5625db5629308b49402306ce4d89430a5371c6 WatchSource:0}: Error finding container d86e67088ad3833f163a70addc5625db5629308b49402306ce4d89430a5371c6: Status 404 returned error can't find the container with id d86e67088ad3833f163a70addc5625db5629308b49402306ce4d89430a5371c6 Jul 23 00:52:42.592632 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:42.592550 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:42.604524 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:42.604492 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" event={"ID":"e5743a3c-5a25-4705-807b-b55376037dc2","Type":"ContainerStarted","Data":"d86e67088ad3833f163a70addc5625db5629308b49402306ce4d89430a5371c6"} Jul 23 00:52:42.695317 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:42.695281 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:43.610768 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:43.610731 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" event={"ID":"e5743a3c-5a25-4705-807b-b55376037dc2","Type":"ContainerStarted","Data":"7183a0e4a03d9c46048425f69706f3da1286747c799bea629374c1422bdacbec"} Jul 23 00:52:43.611165 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:43.610802 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" Jul 23 00:52:43.628298 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:43.628250 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" podStartSLOduration=2.261129706 podStartE2EDuration="2.628234876s" totalImagesPullingTime="367.10517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:52:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:52:43.627493847 +0000 UTC m=+2683.588232467" watchObservedRunningTime="2026-07-23 00:52:43.628234876 +0000 UTC m=+2683.588973493" Jul 23 00:52:46.291209 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:46.291176 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:46.701345 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:46.701310 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:46.966689 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:46.966599 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:47.449280 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:47.449242 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:49.621049 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:49.621025 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" Jul 23 00:52:49.959841 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:49.959810 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:50.065509 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.065474 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:50.075164 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.075125 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql"] Jul 23 00:52:50.079325 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.079303 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" Jul 23 00:52:50.081527 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.081503 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-d29ed6-serving-cert\"" Jul 23 00:52:50.091315 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.091288 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql"] Jul 23 00:52:50.155737 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.155705 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:50.215617 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.215527 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7820a416-ec90-48f3-95a7-f77af3588755-maas-api-tls\") pod \"maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql\" (UID: \"7820a416-ec90-48f3-95a7-f77af3588755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" Jul 23 00:52:50.215617 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.215576 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjrvt\" (UniqueName: \"kubernetes.io/projected/7820a416-ec90-48f3-95a7-f77af3588755-kube-api-access-zjrvt\") pod \"maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql\" (UID: \"7820a416-ec90-48f3-95a7-f77af3588755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" Jul 23 00:52:50.215813 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.215655 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7820a416-ec90-48f3-95a7-f77af3588755-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql\" (UID: \"7820a416-ec90-48f3-95a7-f77af3588755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" Jul 23 00:52:50.257715 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.257685 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:50.316472 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.316433 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjrvt\" (UniqueName: \"kubernetes.io/projected/7820a416-ec90-48f3-95a7-f77af3588755-kube-api-access-zjrvt\") pod \"maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql\" (UID: \"7820a416-ec90-48f3-95a7-f77af3588755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" Jul 23 00:52:50.316675 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.316490 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7820a416-ec90-48f3-95a7-f77af3588755-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql\" (UID: \"7820a416-ec90-48f3-95a7-f77af3588755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" Jul 23 00:52:50.316675 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.316549 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7820a416-ec90-48f3-95a7-f77af3588755-maas-api-tls\") pod \"maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql\" (UID: \"7820a416-ec90-48f3-95a7-f77af3588755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" Jul 23 00:52:50.317174 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.317151 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7820a416-ec90-48f3-95a7-f77af3588755-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql\" (UID: \"7820a416-ec90-48f3-95a7-f77af3588755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" Jul 23 00:52:50.318912 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.318891 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7820a416-ec90-48f3-95a7-f77af3588755-maas-api-tls\") pod \"maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql\" (UID: \"7820a416-ec90-48f3-95a7-f77af3588755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" Jul 23 00:52:50.327729 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.327710 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjrvt\" (UniqueName: \"kubernetes.io/projected/7820a416-ec90-48f3-95a7-f77af3588755-kube-api-access-zjrvt\") pod \"maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql\" (UID: \"7820a416-ec90-48f3-95a7-f77af3588755\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" Jul 23 00:52:50.392754 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.392716 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" Jul 23 00:52:50.534452 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.531469 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql"] Jul 23 00:52:50.643346 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.643314 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" event={"ID":"7820a416-ec90-48f3-95a7-f77af3588755","Type":"ContainerStarted","Data":"665b7e8ec9daa46f0185bcdfc95192d2879f28a3e4ef8b578e29bf16f64f3aa7"} Jul 23 00:52:50.659125 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.659100 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:50.762894 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:50.762799 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:51.650482 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:51.650443 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" event={"ID":"7820a416-ec90-48f3-95a7-f77af3588755","Type":"ContainerStarted","Data":"c5e4525dfa818d31d7b9953bd05b76314442ca950756cb80df7c2ebb4983fce8"} Jul 23 00:52:51.650927 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:51.650573 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" Jul 23 00:52:51.669017 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:51.668970 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" podStartSLOduration=1.254262518 podStartE2EDuration="1.668954656s" totalImagesPullingTime="414.692138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:52:51.667630037 +0000 UTC m=+2691.628368655" watchObservedRunningTime="2026-07-23 00:52:51.668954656 +0000 UTC m=+2691.629693274" Jul 23 00:52:54.456808 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:54.456713 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:52:57.661294 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:52:57.661268 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" Jul 23 00:53:00.588804 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:53:00.588767 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:53:00.606539 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:53:00.606509 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:53:03.509229 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:53:03.509181 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:03.611158 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:53:03.611122 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:04.850980 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:53:04.850953 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:11.565970 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:53:11.565911 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:11.663329 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:53:11.663296 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:48.749543 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:53:48.749475 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:09.674380 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:54:09.674349 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:11.567299 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:54:11.567263 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:24.515244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:54:24.515211 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:36.068132 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:54:36.068101 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:50.413622 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:50.413586 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:51.263637 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:51.263611 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:55.537267 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.537189 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c"] Jul 23 00:55:55.540740 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.540722 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.544490 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.544454 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c75140\"/\"kube-root-ca.crt\"" Jul 23 00:55:55.544490 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.544478 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c75140\"/\"sub-test-model-c75140-kserve-self-signed-certs\"" Jul 23 00:55:55.544658 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.544504 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c75140\"/\"openshift-service-ca.crt\"" Jul 23 00:55:55.544658 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.544573 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c75140\"/\"default-dockercfg-xcq7r\"" Jul 23 00:55:55.544945 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.544914 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-model-cache\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.545030 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.544947 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-tls-certs\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.545030 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.544971 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-home\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.545133 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.545055 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnwc8\" (UniqueName: \"kubernetes.io/projected/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-kube-api-access-nnwc8\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.545133 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.545079 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-dshm\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.545133 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.545113 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-tmp-dir\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.553566 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.553544 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c"] Jul 23 00:55:55.569483 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.569452 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:55.645691 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.645659 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnwc8\" (UniqueName: \"kubernetes.io/projected/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-kube-api-access-nnwc8\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.645865 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.645696 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-dshm\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.645865 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.645730 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-tmp-dir\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.645865 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.645758 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-model-cache\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.645865 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.645780 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-tls-certs\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.645865 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.645798 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-home\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.646218 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.646193 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-model-cache\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.646318 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.646237 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-tmp-dir\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.646318 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.646253 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-home\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.648043 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.648024 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-dshm\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.648214 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.648199 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-tls-certs\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.654404 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.654365 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnwc8\" (UniqueName: \"kubernetes.io/projected/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-kube-api-access-nnwc8\") pod \"sub-test-model-c75140-kserve-76d48c59cf-xkv6c\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.850580 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.850489 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:55:55.990503 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:55.990336 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c"] Jul 23 00:55:55.992847 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:55:55.992822 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6c7065a_d9bc_4a8c_b5ce_df0d8a459e70.slice/crio-7ad28a338f1891bc5b3a1eeb6bab26b394650a0fe45e7ab1cdbfe8d1d3a3317b WatchSource:0}: Error finding container 7ad28a338f1891bc5b3a1eeb6bab26b394650a0fe45e7ab1cdbfe8d1d3a3317b: Status 404 returned error can't find the container with id 7ad28a338f1891bc5b3a1eeb6bab26b394650a0fe45e7ab1cdbfe8d1d3a3317b Jul 23 00:55:56.525154 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:56.525113 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" event={"ID":"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70","Type":"ContainerStarted","Data":"cc81e6d21bc00ce49d63fa2688906233bc8d3afdffb2ebced5482327c3bafa5e"} Jul 23 00:55:56.525334 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:56.525165 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" event={"ID":"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70","Type":"ContainerStarted","Data":"7ad28a338f1891bc5b3a1eeb6bab26b394650a0fe45e7ab1cdbfe8d1d3a3317b"} Jul 23 00:55:56.545092 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:55:56.545038 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" podStartSLOduration=1.5450239030000001 podStartE2EDuration="1.545023903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:55:56.5434351 +0000 UTC m=+2876.504173724" watchObservedRunningTime="2026-07-23 00:55:56.545023903 +0000 UTC m=+2876.505762521" Jul 23 00:56:05.851244 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:56:05.851204 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:56:05.851677 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:56:05.851263 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:56:05.863726 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:56:05.863701 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:56:06.581624 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:56:06.581596 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:56:20.622060 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:56:20.622024 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:05.446270 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:05.446239 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c"] Jul 23 00:57:05.491644 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:05.491616 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:06.376257 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:06.376226 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:14.539089 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:14.539055 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:15.360543 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:15.360494 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:17.111520 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:17.111488 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:17.208738 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:17.208704 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:20.466002 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.465954 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" podUID="a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70" containerName="main" containerID="cri-o://cc81e6d21bc00ce49d63fa2688906233bc8d3afdffb2ebced5482327c3bafa5e" gracePeriod=45 Jul 23 00:57:20.710757 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.710733 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:57:20.817749 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.817641 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-home\") pod \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " Jul 23 00:57:20.817749 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.817724 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-tmp-dir\") pod \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " Jul 23 00:57:20.817992 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.817774 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-model-cache\") pod \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " Jul 23 00:57:20.817992 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.817801 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-dshm\") pod \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " Jul 23 00:57:20.817992 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.817838 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnwc8\" (UniqueName: \"kubernetes.io/projected/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-kube-api-access-nnwc8\") pod \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " Jul 23 00:57:20.817992 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.817885 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-tls-certs\") pod \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\" (UID: \"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70\") " Jul 23 00:57:20.817992 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.817939 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-home" (OuterVolumeSpecName: "home") pod "a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70" (UID: "a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:57:20.818261 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.818003 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70" (UID: "a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:57:20.818261 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.818072 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70" (UID: "a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:57:20.818261 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.818203 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-tmp-dir\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:57:20.818261 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.818220 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-model-cache\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:57:20.818261 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.818228 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-home\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:57:20.820161 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.820134 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-dshm" (OuterVolumeSpecName: "dshm") pod "a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70" (UID: "a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:57:20.820295 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.820168 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70" (UID: "a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:57:20.820295 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.820200 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-kube-api-access-nnwc8" (OuterVolumeSpecName: "kube-api-access-nnwc8") pod "a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70" (UID: "a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70"). InnerVolumeSpecName "kube-api-access-nnwc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:57:20.919059 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.919022 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-dshm\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:57:20.919059 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.919051 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnwc8\" (UniqueName: \"kubernetes.io/projected/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-kube-api-access-nnwc8\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:57:20.919059 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.919064 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70-tls-certs\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:57:20.926210 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.926182 2555 generic.go:356] "Generic (PLEG): container finished" podID="a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70" containerID="cc81e6d21bc00ce49d63fa2688906233bc8d3afdffb2ebced5482327c3bafa5e" exitCode=0 Jul 23 00:57:20.926352 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.926230 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" Jul 23 00:57:20.926352 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.926275 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" event={"ID":"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70","Type":"ContainerDied","Data":"cc81e6d21bc00ce49d63fa2688906233bc8d3afdffb2ebced5482327c3bafa5e"} Jul 23 00:57:20.926352 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.926330 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c" event={"ID":"a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70","Type":"ContainerDied","Data":"7ad28a338f1891bc5b3a1eeb6bab26b394650a0fe45e7ab1cdbfe8d1d3a3317b"} Jul 23 00:57:20.926478 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.926352 2555 scope.go:117] "RemoveContainer" containerID="cc81e6d21bc00ce49d63fa2688906233bc8d3afdffb2ebced5482327c3bafa5e" Jul 23 00:57:20.935969 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.935951 2555 scope.go:117] "RemoveContainer" containerID="cc81e6d21bc00ce49d63fa2688906233bc8d3afdffb2ebced5482327c3bafa5e" Jul 23 00:57:20.936256 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:57:20.936235 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc81e6d21bc00ce49d63fa2688906233bc8d3afdffb2ebced5482327c3bafa5e\": container with ID starting with cc81e6d21bc00ce49d63fa2688906233bc8d3afdffb2ebced5482327c3bafa5e not found: ID does not exist" containerID="cc81e6d21bc00ce49d63fa2688906233bc8d3afdffb2ebced5482327c3bafa5e" Jul 23 00:57:20.936317 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.936270 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc81e6d21bc00ce49d63fa2688906233bc8d3afdffb2ebced5482327c3bafa5e"} err="failed to get container status \"cc81e6d21bc00ce49d63fa2688906233bc8d3afdffb2ebced5482327c3bafa5e\": rpc error: code = NotFound desc = could not find container \"cc81e6d21bc00ce49d63fa2688906233bc8d3afdffb2ebced5482327c3bafa5e\": container with ID starting with cc81e6d21bc00ce49d63fa2688906233bc8d3afdffb2ebced5482327c3bafa5e not found: ID does not exist" Jul 23 00:57:20.947173 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.947143 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c"] Jul 23 00:57:20.951572 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:20.951546 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c75140/sub-test-model-c75140-kserve-76d48c59cf-xkv6c"] Jul 23 00:57:22.502886 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:22.502807 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70" path="/var/lib/kubelet/pods/a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70/volumes" Jul 23 00:57:49.093584 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:49.093527 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:49.183956 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:49.183925 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:54.042551 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.042518 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh"] Jul 23 00:57:54.043046 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.042889 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70" containerName="main" Jul 23 00:57:54.043046 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.042900 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70" containerName="main" Jul 23 00:57:54.043046 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.042972 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6c7065a-d9bc-4a8c-b5ce-df0d8a459e70" containerName="main" Jul 23 00:57:54.046347 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.046321 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.049770 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.049748 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d29ed6\"/\"kube-root-ca.crt\"" Jul 23 00:57:54.049906 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.049748 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d29ed6\"/\"openshift-service-ca.crt\"" Jul 23 00:57:54.049906 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.049747 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d29ed6\"/\"default-dockercfg-fvvtc\"" Jul 23 00:57:54.049906 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.049747 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d29ed6\"/\"rate-test-model-d29ed6-kserve-self-signed-certs\"" Jul 23 00:57:54.054930 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.054902 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh"] Jul 23 00:57:54.076364 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.076323 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:54.112254 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.112217 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnx55\" (UniqueName: \"kubernetes.io/projected/14265e2f-22ad-4839-8718-5e9f57c85707-kube-api-access-pnx55\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.112484 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.112274 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-home\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.112484 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.112369 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-tmp-dir\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.112484 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.112423 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-dshm\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.112484 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.112439 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14265e2f-22ad-4839-8718-5e9f57c85707-tls-certs\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.112728 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.112561 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-model-cache\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.213440 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.213380 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnx55\" (UniqueName: \"kubernetes.io/projected/14265e2f-22ad-4839-8718-5e9f57c85707-kube-api-access-pnx55\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.213631 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.213451 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-home\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.213631 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.213495 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-tmp-dir\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.213631 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.213514 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-dshm\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.213631 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.213529 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14265e2f-22ad-4839-8718-5e9f57c85707-tls-certs\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.213631 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.213597 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-model-cache\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.213923 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.213902 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-tmp-dir\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.213972 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.213901 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-home\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.213972 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.213956 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-model-cache\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.215785 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.215764 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-dshm\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.216157 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.216138 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14265e2f-22ad-4839-8718-5e9f57c85707-tls-certs\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.221857 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.221834 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnx55\" (UniqueName: \"kubernetes.io/projected/14265e2f-22ad-4839-8718-5e9f57c85707-kube-api-access-pnx55\") pod \"rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.357091 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.356997 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:57:54.492549 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:54.492432 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh"] Jul 23 00:57:54.495527 ip-10-0-134-124 kubenswrapper[2555]: W0723 00:57:54.495498 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14265e2f_22ad_4839_8718_5e9f57c85707.slice/crio-b5f8c48f03ba58954b1bb24a4807bff202ccfcdf7d416100c3c64f8799e53b12 WatchSource:0}: Error finding container b5f8c48f03ba58954b1bb24a4807bff202ccfcdf7d416100c3c64f8799e53b12: Status 404 returned error can't find the container with id b5f8c48f03ba58954b1bb24a4807bff202ccfcdf7d416100c3c64f8799e53b12 Jul 23 00:57:55.087721 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:55.087684 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" event={"ID":"14265e2f-22ad-4839-8718-5e9f57c85707","Type":"ContainerStarted","Data":"4f976037e85bc1ffe4e0c6f528bc4bc12d3b1314d6412ebdb0b279d856899610"} Jul 23 00:57:55.087721 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:55.087728 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" event={"ID":"14265e2f-22ad-4839-8718-5e9f57c85707","Type":"ContainerStarted","Data":"b5f8c48f03ba58954b1bb24a4807bff202ccfcdf7d416100c3c64f8799e53b12"} Jul 23 00:57:55.107072 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:57:55.107019 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" podStartSLOduration=1.107002711 podStartE2EDuration="1.107002711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:57:55.104821678 +0000 UTC m=+2995.065560309" watchObservedRunningTime="2026-07-23 00:57:55.107002711 +0000 UTC m=+2995.067741329" Jul 23 00:58:00.620291 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:00.620207 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:58:00.637635 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:00.637607 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 00:58:04.307834 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:04.307795 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:04.357771 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:04.357744 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:58:04.357937 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:04.357792 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:58:04.370656 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:04.370623 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:58:04.395972 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:04.395937 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:05.145608 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:05.145582 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:58:12.527553 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:12.527504 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:12.628106 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:12.628074 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:38.170136 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:38.170099 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:39.038189 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:39.038157 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh"] Jul 23 00:58:39.145050 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:39.145020 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:54.058708 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.058616 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" podUID="14265e2f-22ad-4839-8718-5e9f57c85707" containerName="main" containerID="cri-o://4f976037e85bc1ffe4e0c6f528bc4bc12d3b1314d6412ebdb0b279d856899610" gracePeriod=45 Jul 23 00:58:54.318121 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.318056 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:58:54.369554 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.369500 2555 generic.go:356] "Generic (PLEG): container finished" podID="14265e2f-22ad-4839-8718-5e9f57c85707" containerID="4f976037e85bc1ffe4e0c6f528bc4bc12d3b1314d6412ebdb0b279d856899610" exitCode=0 Jul 23 00:58:54.369554 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.369553 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" Jul 23 00:58:54.369828 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.369589 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" event={"ID":"14265e2f-22ad-4839-8718-5e9f57c85707","Type":"ContainerDied","Data":"4f976037e85bc1ffe4e0c6f528bc4bc12d3b1314d6412ebdb0b279d856899610"} Jul 23 00:58:54.369828 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.369643 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh" event={"ID":"14265e2f-22ad-4839-8718-5e9f57c85707","Type":"ContainerDied","Data":"b5f8c48f03ba58954b1bb24a4807bff202ccfcdf7d416100c3c64f8799e53b12"} Jul 23 00:58:54.369828 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.369666 2555 scope.go:117] "RemoveContainer" containerID="4f976037e85bc1ffe4e0c6f528bc4bc12d3b1314d6412ebdb0b279d856899610" Jul 23 00:58:54.377948 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.377932 2555 scope.go:117] "RemoveContainer" containerID="4f976037e85bc1ffe4e0c6f528bc4bc12d3b1314d6412ebdb0b279d856899610" Jul 23 00:58:54.378201 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:58:54.378182 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f976037e85bc1ffe4e0c6f528bc4bc12d3b1314d6412ebdb0b279d856899610\": container with ID starting with 4f976037e85bc1ffe4e0c6f528bc4bc12d3b1314d6412ebdb0b279d856899610 not found: ID does not exist" containerID="4f976037e85bc1ffe4e0c6f528bc4bc12d3b1314d6412ebdb0b279d856899610" Jul 23 00:58:54.378248 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.378211 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f976037e85bc1ffe4e0c6f528bc4bc12d3b1314d6412ebdb0b279d856899610"} err="failed to get container status \"4f976037e85bc1ffe4e0c6f528bc4bc12d3b1314d6412ebdb0b279d856899610\": rpc error: code = NotFound desc = could not find container \"4f976037e85bc1ffe4e0c6f528bc4bc12d3b1314d6412ebdb0b279d856899610\": container with ID starting with 4f976037e85bc1ffe4e0c6f528bc4bc12d3b1314d6412ebdb0b279d856899610 not found: ID does not exist" Jul 23 00:58:54.475886 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.475849 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-dshm\") pod \"14265e2f-22ad-4839-8718-5e9f57c85707\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " Jul 23 00:58:54.476067 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.475939 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnx55\" (UniqueName: \"kubernetes.io/projected/14265e2f-22ad-4839-8718-5e9f57c85707-kube-api-access-pnx55\") pod \"14265e2f-22ad-4839-8718-5e9f57c85707\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " Jul 23 00:58:54.476067 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.475973 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14265e2f-22ad-4839-8718-5e9f57c85707-tls-certs\") pod \"14265e2f-22ad-4839-8718-5e9f57c85707\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " Jul 23 00:58:54.476067 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.476007 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-home\") pod \"14265e2f-22ad-4839-8718-5e9f57c85707\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " Jul 23 00:58:54.476067 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.476052 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-model-cache\") pod \"14265e2f-22ad-4839-8718-5e9f57c85707\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " Jul 23 00:58:54.476288 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.476093 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-tmp-dir\") pod \"14265e2f-22ad-4839-8718-5e9f57c85707\" (UID: \"14265e2f-22ad-4839-8718-5e9f57c85707\") " Jul 23 00:58:54.476288 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.476267 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-model-cache" (OuterVolumeSpecName: "model-cache") pod "14265e2f-22ad-4839-8718-5e9f57c85707" (UID: "14265e2f-22ad-4839-8718-5e9f57c85707"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:58:54.476411 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.476283 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-home" (OuterVolumeSpecName: "home") pod "14265e2f-22ad-4839-8718-5e9f57c85707" (UID: "14265e2f-22ad-4839-8718-5e9f57c85707"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:58:54.476411 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.476373 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14265e2f-22ad-4839-8718-5e9f57c85707" (UID: "14265e2f-22ad-4839-8718-5e9f57c85707"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:58:54.476507 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.476438 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-home\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:58:54.476507 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.476456 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-model-cache\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:58:54.478225 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.478204 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14265e2f-22ad-4839-8718-5e9f57c85707-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14265e2f-22ad-4839-8718-5e9f57c85707" (UID: "14265e2f-22ad-4839-8718-5e9f57c85707"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:58:54.478486 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.478465 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-dshm" (OuterVolumeSpecName: "dshm") pod "14265e2f-22ad-4839-8718-5e9f57c85707" (UID: "14265e2f-22ad-4839-8718-5e9f57c85707"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:58:54.478623 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.478604 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14265e2f-22ad-4839-8718-5e9f57c85707-kube-api-access-pnx55" (OuterVolumeSpecName: "kube-api-access-pnx55") pod "14265e2f-22ad-4839-8718-5e9f57c85707" (UID: "14265e2f-22ad-4839-8718-5e9f57c85707"). InnerVolumeSpecName "kube-api-access-pnx55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:58:54.577883 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.577797 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14265e2f-22ad-4839-8718-5e9f57c85707-tls-certs\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:58:54.577883 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.577848 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-tmp-dir\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:58:54.577883 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.577857 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14265e2f-22ad-4839-8718-5e9f57c85707-dshm\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:58:54.577883 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.577866 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnx55\" (UniqueName: \"kubernetes.io/projected/14265e2f-22ad-4839-8718-5e9f57c85707-kube-api-access-pnx55\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:58:54.698330 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.698291 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh"] Jul 23 00:58:54.710998 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:54.710972 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/rate-test-model-d29ed6-kserve-86cdc94d67-pz7sh"] Jul 23 00:58:56.502412 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:56.502361 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14265e2f-22ad-4839-8718-5e9f57c85707" path="/var/lib/kubelet/pods/14265e2f-22ad-4839-8718-5e9f57c85707/volumes" Jul 23 00:58:58.751470 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:58.751433 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:58.852910 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:58:58.852876 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:08.485256 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:08.485224 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72"] Jul 23 00:59:08.485765 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:08.485488 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" podUID="e5743a3c-5a25-4705-807b-b55376037dc2" containerName="maas-api" containerID="cri-o://7183a0e4a03d9c46048425f69706f3da1286747c799bea629374c1422bdacbec" gracePeriod=30 Jul 23 00:59:08.512901 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:08.512870 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:08.926318 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:08.926287 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" Jul 23 00:59:09.109648 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.109563 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5743a3c-5a25-4705-807b-b55376037dc2-openshift-service-ca\") pod \"e5743a3c-5a25-4705-807b-b55376037dc2\" (UID: \"e5743a3c-5a25-4705-807b-b55376037dc2\") " Jul 23 00:59:09.109648 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.109611 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5743a3c-5a25-4705-807b-b55376037dc2-maas-api-tls\") pod \"e5743a3c-5a25-4705-807b-b55376037dc2\" (UID: \"e5743a3c-5a25-4705-807b-b55376037dc2\") " Jul 23 00:59:09.109933 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.109700 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sdcr\" (UniqueName: \"kubernetes.io/projected/e5743a3c-5a25-4705-807b-b55376037dc2-kube-api-access-2sdcr\") pod \"e5743a3c-5a25-4705-807b-b55376037dc2\" (UID: \"e5743a3c-5a25-4705-807b-b55376037dc2\") " Jul 23 00:59:09.109933 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.109899 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5743a3c-5a25-4705-807b-b55376037dc2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e5743a3c-5a25-4705-807b-b55376037dc2" (UID: "e5743a3c-5a25-4705-807b-b55376037dc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:59:09.110017 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.109995 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5743a3c-5a25-4705-807b-b55376037dc2-openshift-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:59:09.111768 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.111746 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5743a3c-5a25-4705-807b-b55376037dc2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e5743a3c-5a25-4705-807b-b55376037dc2" (UID: "e5743a3c-5a25-4705-807b-b55376037dc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:59:09.111828 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.111789 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5743a3c-5a25-4705-807b-b55376037dc2-kube-api-access-2sdcr" (OuterVolumeSpecName: "kube-api-access-2sdcr") pod "e5743a3c-5a25-4705-807b-b55376037dc2" (UID: "e5743a3c-5a25-4705-807b-b55376037dc2"). InnerVolumeSpecName "kube-api-access-2sdcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:59:09.211246 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.211210 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sdcr\" (UniqueName: \"kubernetes.io/projected/e5743a3c-5a25-4705-807b-b55376037dc2-kube-api-access-2sdcr\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:59:09.211246 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.211239 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5743a3c-5a25-4705-807b-b55376037dc2-maas-api-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:59:09.440904 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.440874 2555 generic.go:356] "Generic (PLEG): container finished" podID="e5743a3c-5a25-4705-807b-b55376037dc2" containerID="7183a0e4a03d9c46048425f69706f3da1286747c799bea629374c1422bdacbec" exitCode=0 Jul 23 00:59:09.441071 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.440921 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" Jul 23 00:59:09.441071 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.440954 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" event={"ID":"e5743a3c-5a25-4705-807b-b55376037dc2","Type":"ContainerDied","Data":"7183a0e4a03d9c46048425f69706f3da1286747c799bea629374c1422bdacbec"} Jul 23 00:59:09.441071 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.441003 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72" event={"ID":"e5743a3c-5a25-4705-807b-b55376037dc2","Type":"ContainerDied","Data":"d86e67088ad3833f163a70addc5625db5629308b49402306ce4d89430a5371c6"} Jul 23 00:59:09.441071 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.441027 2555 scope.go:117] "RemoveContainer" containerID="7183a0e4a03d9c46048425f69706f3da1286747c799bea629374c1422bdacbec" Jul 23 00:59:09.450707 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.450693 2555 scope.go:117] "RemoveContainer" containerID="7183a0e4a03d9c46048425f69706f3da1286747c799bea629374c1422bdacbec" Jul 23 00:59:09.450970 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:59:09.450953 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7183a0e4a03d9c46048425f69706f3da1286747c799bea629374c1422bdacbec\": container with ID starting with 7183a0e4a03d9c46048425f69706f3da1286747c799bea629374c1422bdacbec not found: ID does not exist" containerID="7183a0e4a03d9c46048425f69706f3da1286747c799bea629374c1422bdacbec" Jul 23 00:59:09.451021 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.450979 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7183a0e4a03d9c46048425f69706f3da1286747c799bea629374c1422bdacbec"} err="failed to get container status \"7183a0e4a03d9c46048425f69706f3da1286747c799bea629374c1422bdacbec\": rpc error: code = NotFound desc = could not find container \"7183a0e4a03d9c46048425f69706f3da1286747c799bea629374c1422bdacbec\": container with ID starting with 7183a0e4a03d9c46048425f69706f3da1286747c799bea629374c1422bdacbec not found: ID does not exist" Jul 23 00:59:09.464204 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.464172 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72"] Jul 23 00:59:09.469199 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:09.469179 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c75140-7f4bbdc66f-5xb72"] Jul 23 00:59:10.507700 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:10.507668 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5743a3c-5a25-4705-807b-b55376037dc2" path="/var/lib/kubelet/pods/e5743a3c-5a25-4705-807b-b55376037dc2/volumes" Jul 23 00:59:13.151062 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:13.151031 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:14.648197 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:14.648159 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:19.763145 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:19.763117 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:19.865900 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:19.865873 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:20.060594 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:20.060522 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:20.300123 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:20.300093 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:21.204049 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:21.204018 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:21.308615 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:21.308587 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:31.094860 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.094829 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql"] Jul 23 00:59:31.095249 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.095033 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" podUID="7820a416-ec90-48f3-95a7-f77af3588755" containerName="maas-api" containerID="cri-o://c5e4525dfa818d31d7b9953bd05b76314442ca950756cb80df7c2ebb4983fce8" gracePeriod=30 Jul 23 00:59:31.124270 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.124245 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:31.353225 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.353163 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" Jul 23 00:59:31.407302 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.407272 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjrvt\" (UniqueName: \"kubernetes.io/projected/7820a416-ec90-48f3-95a7-f77af3588755-kube-api-access-zjrvt\") pod \"7820a416-ec90-48f3-95a7-f77af3588755\" (UID: \"7820a416-ec90-48f3-95a7-f77af3588755\") " Jul 23 00:59:31.407473 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.407340 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7820a416-ec90-48f3-95a7-f77af3588755-maas-api-tls\") pod \"7820a416-ec90-48f3-95a7-f77af3588755\" (UID: \"7820a416-ec90-48f3-95a7-f77af3588755\") " Jul 23 00:59:31.407473 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.407361 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7820a416-ec90-48f3-95a7-f77af3588755-openshift-service-ca\") pod \"7820a416-ec90-48f3-95a7-f77af3588755\" (UID: \"7820a416-ec90-48f3-95a7-f77af3588755\") " Jul 23 00:59:31.407789 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.407757 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7820a416-ec90-48f3-95a7-f77af3588755-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7820a416-ec90-48f3-95a7-f77af3588755" (UID: "7820a416-ec90-48f3-95a7-f77af3588755"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:59:31.409318 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.409290 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7820a416-ec90-48f3-95a7-f77af3588755-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7820a416-ec90-48f3-95a7-f77af3588755" (UID: "7820a416-ec90-48f3-95a7-f77af3588755"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:59:31.409429 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.409344 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7820a416-ec90-48f3-95a7-f77af3588755-kube-api-access-zjrvt" (OuterVolumeSpecName: "kube-api-access-zjrvt") pod "7820a416-ec90-48f3-95a7-f77af3588755" (UID: "7820a416-ec90-48f3-95a7-f77af3588755"). InnerVolumeSpecName "kube-api-access-zjrvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:59:31.508063 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.508031 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjrvt\" (UniqueName: \"kubernetes.io/projected/7820a416-ec90-48f3-95a7-f77af3588755-kube-api-access-zjrvt\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:59:31.508063 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.508059 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7820a416-ec90-48f3-95a7-f77af3588755-maas-api-tls\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:59:31.508063 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.508068 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7820a416-ec90-48f3-95a7-f77af3588755-openshift-service-ca\") on node \"ip-10-0-134-124.ec2.internal\" DevicePath \"\"" Jul 23 00:59:31.545102 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.545069 2555 generic.go:356] "Generic (PLEG): container finished" podID="7820a416-ec90-48f3-95a7-f77af3588755" containerID="c5e4525dfa818d31d7b9953bd05b76314442ca950756cb80df7c2ebb4983fce8" exitCode=0 Jul 23 00:59:31.545233 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.545122 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" Jul 23 00:59:31.545233 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.545151 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" event={"ID":"7820a416-ec90-48f3-95a7-f77af3588755","Type":"ContainerDied","Data":"c5e4525dfa818d31d7b9953bd05b76314442ca950756cb80df7c2ebb4983fce8"} Jul 23 00:59:31.545233 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.545207 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql" event={"ID":"7820a416-ec90-48f3-95a7-f77af3588755","Type":"ContainerDied","Data":"665b7e8ec9daa46f0185bcdfc95192d2879f28a3e4ef8b578e29bf16f64f3aa7"} Jul 23 00:59:31.545233 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.545228 2555 scope.go:117] "RemoveContainer" containerID="c5e4525dfa818d31d7b9953bd05b76314442ca950756cb80df7c2ebb4983fce8" Jul 23 00:59:31.554231 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.554213 2555 scope.go:117] "RemoveContainer" containerID="c5e4525dfa818d31d7b9953bd05b76314442ca950756cb80df7c2ebb4983fce8" Jul 23 00:59:31.554532 ip-10-0-134-124 kubenswrapper[2555]: E0723 00:59:31.554505 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e4525dfa818d31d7b9953bd05b76314442ca950756cb80df7c2ebb4983fce8\": container with ID starting with c5e4525dfa818d31d7b9953bd05b76314442ca950756cb80df7c2ebb4983fce8 not found: ID does not exist" containerID="c5e4525dfa818d31d7b9953bd05b76314442ca950756cb80df7c2ebb4983fce8" Jul 23 00:59:31.554586 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.554541 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e4525dfa818d31d7b9953bd05b76314442ca950756cb80df7c2ebb4983fce8"} err="failed to get container status \"c5e4525dfa818d31d7b9953bd05b76314442ca950756cb80df7c2ebb4983fce8\": rpc error: code = NotFound desc = could not find container \"c5e4525dfa818d31d7b9953bd05b76314442ca950756cb80df7c2ebb4983fce8\": container with ID starting with c5e4525dfa818d31d7b9953bd05b76314442ca950756cb80df7c2ebb4983fce8 not found: ID does not exist" Jul 23 00:59:31.566653 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.566628 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql"] Jul 23 00:59:31.571061 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:31.571041 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d29ed6-59bb4849b-z89ql"] Jul 23 00:59:32.502522 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:32.502491 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7820a416-ec90-48f3-95a7-f77af3588755" path="/var/lib/kubelet/pods/7820a416-ec90-48f3-95a7-f77af3588755/volumes" Jul 23 00:59:35.508313 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:35.508281 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:35.898677 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:35.898593 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:41.787112 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:41.787077 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:41.888675 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:41.888630 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:42.159612 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:42.159590 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:42.329861 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:42.329827 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:42.742180 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:42.742147 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86d8db559f-zjzj8_2a881541-ed40-44b0-8709-2e1af5c1204e/authorino/0.log" Jul 23 00:59:49.200967 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:49.200943 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2_322c84c8-b4a2-4e9f-983a-84b63db49538/copy-manifests/0.log" Jul 23 00:59:49.208905 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:49.208881 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2_322c84c8-b4a2-4e9f-983a-84b63db49538/manager/0.log" Jul 23 00:59:49.569711 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:49.569637 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-b877fd55f-hzg9d_717116c9-0961-46e5-bff6-125cb9a7d4db/maas-api/0.log" Jul 23 00:59:51.389045 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:51.389018 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86d8db559f-zjzj8_2a881541-ed40-44b0-8709-2e1af5c1204e/authorino/0.log" Jul 23 00:59:52.196110 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:52.196074 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ggwgh_f748a5da-aa5e-4904-8c16-abe0232e1045/manager/0.log" Jul 23 00:59:52.932185 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:52.932155 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7447dc7bcf-qlpg6_76a01cbd-db3e-4851-8306-0db08ea7e428/kube-auth-proxy/0.log" Jul 23 00:59:54.200768 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:54.200735 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv2shk_65aa11a5-cb74-4e56-ac70-22333c42f4a9/main/0.log" Jul 23 00:59:54.319323 ip-10-0-134-124 kubenswrapper[2555]: I0723 00:59:54.319298 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-dqmfd_ab76740d-0364-4782-8c64-b1ecacd62265/main/0.log" Jul 23 01:00:00.396249 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:00.396220 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:00.488813 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:00.488774 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:03.425018 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:03.424982 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jbwvd_5981d2c3-3af8-4f97-b00f-eed942743c86/global-pull-secret-syncer/0.log" Jul 23 01:00:03.515420 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:03.515376 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2x8c9_76c4a8e8-f0b1-4468-996d-dae6883fdca4/konnectivity-agent/0.log" Jul 23 01:00:03.633331 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:03.633292 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-124.ec2.internal_6c386b32f5b824bf53480b029cd79f9a/haproxy/0.log" Jul 23 01:00:08.935996 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:08.935963 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86d8db559f-zjzj8_2a881541-ed40-44b0-8709-2e1af5c1204e/authorino/0.log" Jul 23 01:00:09.399499 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:09.399468 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ggwgh_f748a5da-aa5e-4904-8c16-abe0232e1045/manager/0.log" Jul 23 01:00:11.200129 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:11.200079 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-qm5ml_165778da-7482-49cf-948f-5440431f12e1/cluster-monitoring-operator/0.log" Jul 23 01:00:11.229880 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:11.229849 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-lmdpb_8db7756f-5d48-404c-9529-4601b1fedbd0/kube-state-metrics/0.log" Jul 23 01:00:11.279767 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:11.279718 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-lmdpb_8db7756f-5d48-404c-9529-4601b1fedbd0/kube-rbac-proxy-main/0.log" Jul 23 01:00:11.405448 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:11.405357 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-lmdpb_8db7756f-5d48-404c-9529-4601b1fedbd0/kube-rbac-proxy-self/0.log" Jul 23 01:00:11.677686 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:11.677660 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8jgfv_722842b0-0473-4e43-bbd2-fed93e985695/node-exporter/0.log" Jul 23 01:00:11.699130 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:11.699106 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8jgfv_722842b0-0473-4e43-bbd2-fed93e985695/kube-rbac-proxy/0.log" Jul 23 01:00:11.720698 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:11.720673 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8jgfv_722842b0-0473-4e43-bbd2-fed93e985695/init-textfile/0.log" Jul 23 01:00:11.832683 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:11.832655 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-d56b4_111a2305-1c97-4e01-907e-eeef54a558bb/kube-rbac-proxy-main/0.log" Jul 23 01:00:11.856218 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:11.856193 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-d56b4_111a2305-1c97-4e01-907e-eeef54a558bb/kube-rbac-proxy-self/0.log" Jul 23 01:00:11.879622 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:11.879586 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-d56b4_111a2305-1c97-4e01-907e-eeef54a558bb/openshift-state-metrics/0.log" Jul 23 01:00:11.918546 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:11.918521 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eeededa1-ed50-413b-b0cd-08e0eb02d7ee/prometheus/0.log" Jul 23 01:00:11.941071 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:11.940993 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eeededa1-ed50-413b-b0cd-08e0eb02d7ee/config-reloader/0.log" Jul 23 01:00:11.965283 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:11.965253 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eeededa1-ed50-413b-b0cd-08e0eb02d7ee/thanos-sidecar/0.log" Jul 23 01:00:11.987767 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:11.987736 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eeededa1-ed50-413b-b0cd-08e0eb02d7ee/kube-rbac-proxy-web/0.log" Jul 23 01:00:12.013671 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:12.013642 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eeededa1-ed50-413b-b0cd-08e0eb02d7ee/kube-rbac-proxy/0.log" Jul 23 01:00:12.039112 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:12.039082 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eeededa1-ed50-413b-b0cd-08e0eb02d7ee/kube-rbac-proxy-thanos/0.log" Jul 23 01:00:12.063512 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:12.063477 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eeededa1-ed50-413b-b0cd-08e0eb02d7ee/init-config-reloader/0.log" Jul 23 01:00:12.095982 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:12.095956 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-jfqbc_457d585b-f5fe-4bd7-a933-c958d30917ab/prometheus-operator/0.log" Jul 23 01:00:12.135171 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:12.135142 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-jfqbc_457d585b-f5fe-4bd7-a933-c958d30917ab/kube-rbac-proxy/0.log" Jul 23 01:00:12.173311 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:12.173286 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-sg5wc_ef62896d-92ba-4312-bbf5-dcb3b85131eb/prometheus-operator-admission-webhook/0.log" Jul 23 01:00:13.501957 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.501920 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m"] Jul 23 01:00:13.502451 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.502254 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5743a3c-5a25-4705-807b-b55376037dc2" containerName="maas-api" Jul 23 01:00:13.502451 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.502264 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5743a3c-5a25-4705-807b-b55376037dc2" containerName="maas-api" Jul 23 01:00:13.502451 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.502280 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7820a416-ec90-48f3-95a7-f77af3588755" containerName="maas-api" Jul 23 01:00:13.502451 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.502294 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7820a416-ec90-48f3-95a7-f77af3588755" containerName="maas-api" Jul 23 01:00:13.502451 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.502308 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14265e2f-22ad-4839-8718-5e9f57c85707" containerName="main" Jul 23 01:00:13.502451 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.502314 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="14265e2f-22ad-4839-8718-5e9f57c85707" containerName="main" Jul 23 01:00:13.502451 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.502373 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7820a416-ec90-48f3-95a7-f77af3588755" containerName="maas-api" Jul 23 01:00:13.502451 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.502379 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="14265e2f-22ad-4839-8718-5e9f57c85707" containerName="main" Jul 23 01:00:13.502451 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.502385 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5743a3c-5a25-4705-807b-b55376037dc2" containerName="maas-api" Jul 23 01:00:13.505452 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.505436 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:13.508463 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.508444 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mtxqj\"/\"kube-root-ca.crt\"" Jul 23 01:00:13.509230 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.509213 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mtxqj\"/\"openshift-service-ca.crt\"" Jul 23 01:00:13.509288 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.509219 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mtxqj\"/\"default-dockercfg-t6rqd\"" Jul 23 01:00:13.514835 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.514807 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m"] Jul 23 01:00:13.573689 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.573661 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/72cc49dc-4925-460b-a336-dbe8dcd28185-proc\") pod \"perf-node-gather-daemonset-cxb4m\" (UID: \"72cc49dc-4925-460b-a336-dbe8dcd28185\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:13.573689 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.573692 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/72cc49dc-4925-460b-a336-dbe8dcd28185-podres\") pod \"perf-node-gather-daemonset-cxb4m\" (UID: \"72cc49dc-4925-460b-a336-dbe8dcd28185\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:13.573865 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.573778 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72cc49dc-4925-460b-a336-dbe8dcd28185-sys\") pod \"perf-node-gather-daemonset-cxb4m\" (UID: \"72cc49dc-4925-460b-a336-dbe8dcd28185\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:13.573865 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.573804 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlsjt\" (UniqueName: \"kubernetes.io/projected/72cc49dc-4925-460b-a336-dbe8dcd28185-kube-api-access-mlsjt\") pod \"perf-node-gather-daemonset-cxb4m\" (UID: \"72cc49dc-4925-460b-a336-dbe8dcd28185\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:13.573865 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.573848 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72cc49dc-4925-460b-a336-dbe8dcd28185-lib-modules\") pod \"perf-node-gather-daemonset-cxb4m\" (UID: \"72cc49dc-4925-460b-a336-dbe8dcd28185\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:13.678249 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.678219 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/72cc49dc-4925-460b-a336-dbe8dcd28185-proc\") pod \"perf-node-gather-daemonset-cxb4m\" (UID: \"72cc49dc-4925-460b-a336-dbe8dcd28185\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:13.678436 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.678252 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/72cc49dc-4925-460b-a336-dbe8dcd28185-podres\") pod \"perf-node-gather-daemonset-cxb4m\" (UID: \"72cc49dc-4925-460b-a336-dbe8dcd28185\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:13.678436 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.678426 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/72cc49dc-4925-460b-a336-dbe8dcd28185-proc\") pod \"perf-node-gather-daemonset-cxb4m\" (UID: \"72cc49dc-4925-460b-a336-dbe8dcd28185\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:13.678537 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.678497 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/72cc49dc-4925-460b-a336-dbe8dcd28185-podres\") pod \"perf-node-gather-daemonset-cxb4m\" (UID: \"72cc49dc-4925-460b-a336-dbe8dcd28185\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:13.678537 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.678502 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72cc49dc-4925-460b-a336-dbe8dcd28185-sys\") pod \"perf-node-gather-daemonset-cxb4m\" (UID: \"72cc49dc-4925-460b-a336-dbe8dcd28185\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:13.678612 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.678539 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlsjt\" (UniqueName: \"kubernetes.io/projected/72cc49dc-4925-460b-a336-dbe8dcd28185-kube-api-access-mlsjt\") pod \"perf-node-gather-daemonset-cxb4m\" (UID: \"72cc49dc-4925-460b-a336-dbe8dcd28185\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:13.678612 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.678564 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72cc49dc-4925-460b-a336-dbe8dcd28185-sys\") pod \"perf-node-gather-daemonset-cxb4m\" (UID: \"72cc49dc-4925-460b-a336-dbe8dcd28185\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:13.678612 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.678599 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72cc49dc-4925-460b-a336-dbe8dcd28185-lib-modules\") pod \"perf-node-gather-daemonset-cxb4m\" (UID: \"72cc49dc-4925-460b-a336-dbe8dcd28185\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:13.678766 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.678750 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72cc49dc-4925-460b-a336-dbe8dcd28185-lib-modules\") pod \"perf-node-gather-daemonset-cxb4m\" (UID: \"72cc49dc-4925-460b-a336-dbe8dcd28185\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:13.687300 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.687280 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlsjt\" (UniqueName: \"kubernetes.io/projected/72cc49dc-4925-460b-a336-dbe8dcd28185-kube-api-access-mlsjt\") pod \"perf-node-gather-daemonset-cxb4m\" (UID: \"72cc49dc-4925-460b-a336-dbe8dcd28185\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:13.815966 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.815865 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:13.946801 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:13.946775 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m"] Jul 23 01:00:13.947959 ip-10-0-134-124 kubenswrapper[2555]: W0723 01:00:13.947937 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod72cc49dc_4925_460b_a336_dbe8dcd28185.slice/crio-8b04d9292ace33b8a9bc2d52eee444ee67a303588cacba9f8a05309557efecbe WatchSource:0}: Error finding container 8b04d9292ace33b8a9bc2d52eee444ee67a303588cacba9f8a05309557efecbe: Status 404 returned error can't find the container with id 8b04d9292ace33b8a9bc2d52eee444ee67a303588cacba9f8a05309557efecbe Jul 23 01:00:14.222621 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:14.222592 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/1.log" Jul 23 01:00:14.228025 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:14.228005 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-98rlm_19897756-2813-4980-865c-1bbb2012aa33/console-operator/2.log" Jul 23 01:00:14.688587 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:14.688560 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d64c7cf85-l4nl5_4a71497f-0c0d-46ec-ab70-2805745ea5c4/console/0.log" Jul 23 01:00:14.722849 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:14.722824 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-8cg64_4faf92ae-a088-43b4-a83f-ab0cab855e27/download-server/0.log" Jul 23 01:00:14.743047 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:14.743021 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" event={"ID":"72cc49dc-4925-460b-a336-dbe8dcd28185","Type":"ContainerStarted","Data":"ad158c5673f2d2e596c28b5d75f60303dc0134faa71aa0fc4cb83f143a0b5de5"} Jul 23 01:00:14.743178 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:14.743067 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" event={"ID":"72cc49dc-4925-460b-a336-dbe8dcd28185","Type":"ContainerStarted","Data":"8b04d9292ace33b8a9bc2d52eee444ee67a303588cacba9f8a05309557efecbe"} Jul 23 01:00:14.743178 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:14.743097 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:14.760801 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:14.760729 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" podStartSLOduration=1.760714336 podStartE2EDuration="1.760714336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:00:14.759158875 +0000 UTC m=+3134.719897494" watchObservedRunningTime="2026-07-23 01:00:14.760714336 +0000 UTC m=+3134.721452955" Jul 23 01:00:16.495476 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:16.495445 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qwwgq_359cf272-977d-4bcd-8e31-d2d45a8b440f/dns/0.log" Jul 23 01:00:16.517741 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:16.517718 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qwwgq_359cf272-977d-4bcd-8e31-d2d45a8b440f/kube-rbac-proxy/0.log" Jul 23 01:00:16.653306 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:16.653273 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t75zr_225d2d8c-3a75-47f8-9943-2f67e3c62e8b/dns-node-resolver/0.log" Jul 23 01:00:17.310810 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:17.310776 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pn5jp_754c2d81-3f3e-44d4-8589-fec36d904b15/node-ca/0.log" Jul 23 01:00:18.467873 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:18.467844 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7447dc7bcf-qlpg6_76a01cbd-db3e-4851-8306-0db08ea7e428/kube-auth-proxy/0.log" Jul 23 01:00:19.286026 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:19.285998 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f5wg8_5e4b524a-e6ce-4247-bf26-81fda530d64d/serve-healthcheck-canary/0.log" Jul 23 01:00:19.924048 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:19.924022 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9db4t_1c3cef2c-367d-4cea-bf38-e952f424ff69/kube-rbac-proxy/0.log" Jul 23 01:00:19.947252 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:19.947222 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9db4t_1c3cef2c-367d-4cea-bf38-e952f424ff69/exporter/0.log" Jul 23 01:00:19.972180 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:19.972141 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9db4t_1c3cef2c-367d-4cea-bf38-e952f424ff69/extractor/0.log" Jul 23 01:00:20.757554 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:20.757521 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-cxb4m" Jul 23 01:00:23.002922 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:23.002896 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2_322c84c8-b4a2-4e9f-983a-84b63db49538/manager/0.log" Jul 23 01:00:23.027518 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:23.027494 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f65bfdcb7-sdcv2_322c84c8-b4a2-4e9f-983a-84b63db49538/copy-manifests/0.log" Jul 23 01:00:23.702823 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:23.702799 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7447dc7bcf-qlpg6_76a01cbd-db3e-4851-8306-0db08ea7e428/kube-auth-proxy/0.log" Jul 23 01:00:24.310794 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:24.310751 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-85d467fcb7-86m4g_ff067220-60af-4687-b2e3-613d56520cfb/manager/0.log" Jul 23 01:00:25.161424 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:25.161374 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7447dc7bcf-qlpg6_76a01cbd-db3e-4851-8306-0db08ea7e428/kube-auth-proxy/0.log" Jul 23 01:00:29.736717 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:29.736680 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-r7vwm_73b367b0-9408-4fcd-9535-cb198cfd7e34/kube-storage-version-migrator-operator/0.log" Jul 23 01:00:29.740170 ip-10-0-134-124 kubenswrapper[2555]: I0723 01:00:29.740120 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-r7vwm_73b367b0-9408-4fcd-9535-cb198cfd7e34/kube-storage-version-migrator-operator/1.log"