Jul 14 21:35:27.988981 ip-10-0-130-223 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 14 21:35:27.988991 ip-10-0-130-223 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 14 21:35:27.988998 ip-10-0-130-223 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 14 21:35:27.989227 ip-10-0-130-223 systemd[1]: Failed to start Kubernetes Kubelet. Jul 14 21:35:38.162536 ip-10-0-130-223 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 14 21:35:38.162549 ip-10-0-130-223 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 34c98588e8544f40b5a1afb1d4423d6e -- Jul 14 21:37:13.328477 ip-10-0-130-223 systemd[1]: Starting Kubernetes Kubelet... Jul 14 21:37:13.899301 ip-10-0-130-223 kubenswrapper[2552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 21:37:13.899301 ip-10-0-130-223 kubenswrapper[2552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 21:37:13.899301 ip-10-0-130-223 kubenswrapper[2552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 21:37:13.899301 ip-10-0-130-223 kubenswrapper[2552]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 21:37:13.899301 ip-10-0-130-223 kubenswrapper[2552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 21:37:13.899301 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.780055 2552 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 21:37:13.899301 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784868 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 21:37:13.899301 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784883 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 21:37:13.899301 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784887 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 21:37:13.899301 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784891 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 21:37:13.899301 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784894 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 21:37:13.899301 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784897 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 21:37:13.899301 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784900 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 21:37:13.899301 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784903 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 21:37:13.869514 ip-10-0-130-223 systemd[1]: Started Kubernetes Kubelet. Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784906 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784910 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784913 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784915 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784918 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784921 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784924 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784926 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784929 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784931 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784936 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784940 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784943 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784947 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784950 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784953 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784956 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784965 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784968 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 21:37:13.901923 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784971 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784974 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784977 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784979 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784982 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784985 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784987 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784990 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784993 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784996 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.784998 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785001 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785004 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785006 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785009 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785012 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785014 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785017 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785021 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 21:37:13.916331 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785025 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785028 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785031 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785035 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785037 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785041 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785043 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785046 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785048 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785052 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785054 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785057 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785060 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785063 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785065 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785068 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785071 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785074 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785076 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785079 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 21:37:13.931656 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785081 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785084 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785087 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785089 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785092 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785094 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785097 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785100 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785102 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785105 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785107 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785109 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785113 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785116 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785118 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785121 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785123 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785129 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785132 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 21:37:13.947415 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785134 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785137 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785139 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785142 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785145 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785148 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785150 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785155 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785159 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785162 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785164 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785167 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785170 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.785173 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787469 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787478 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787481 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787485 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787488 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787491 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 21:37:13.963575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787494 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787496 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787499 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787503 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787507 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787510 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787513 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787515 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787519 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787522 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787527 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787531 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787535 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787538 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787541 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787544 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787546 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787550 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787553 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 21:37:13.979379 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787555 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787558 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787560 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787563 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787565 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787568 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787570 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787573 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787575 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787578 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787582 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787585 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787588 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787591 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787593 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787596 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787599 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787601 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787604 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787606 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 21:37:14.116088 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787609 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787611 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787614 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787620 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787623 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787625 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787628 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787630 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787633 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787635 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787638 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787641 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787644 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787646 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787649 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787651 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787654 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787656 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787659 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787661 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 21:37:14.117608 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787664 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787666 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787670 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787673 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787675 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787678 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787683 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787686 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787690 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787692 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787695 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787698 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787701 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787703 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787705 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787710 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787713 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787715 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787718 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787720 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 21:37:14.118572 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787722 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787725 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787727 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787730 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787732 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787735 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787737 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787740 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787742 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787745 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787747 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787749 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787752 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.787755 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788149 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788154 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788156 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788159 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788171 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 21:37:14.119169 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788175 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788179 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788182 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788184 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788202 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788206 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788208 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788211 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788215 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788220 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788223 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788227 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788230 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788233 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788235 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788239 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788241 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788244 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788247 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788249 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 21:37:14.119793 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788252 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788255 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788258 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788261 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788264 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788267 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788270 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788274 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788277 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788279 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788282 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788284 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788293 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788296 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788298 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788301 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788303 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788306 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788308 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788311 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 21:37:14.120394 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788315 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788317 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788320 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788322 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788325 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788329 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788332 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788335 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788337 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788340 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788343 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788345 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788347 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788350 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788352 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788355 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788357 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788360 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788362 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788366 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 21:37:14.120985 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788368 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788371 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788373 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788376 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788378 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788387 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788391 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788394 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788396 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788399 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788401 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788404 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788407 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788410 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788412 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788415 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788417 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788420 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788422 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788424 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 21:37:14.121603 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788427 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788430 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788432 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788435 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788438 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788441 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788443 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788446 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788454 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788457 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788460 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788463 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788466 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.788469 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788544 2552 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788552 2552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788572 2552 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788576 2552 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788587 2552 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788590 2552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788594 2552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 21:37:14.122213 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788599 2552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788602 2552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788605 2552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788610 2552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788613 2552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788616 2552 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788619 2552 flags.go:64] FLAG: --cgroup-root="" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788622 2552 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788625 2552 flags.go:64] FLAG: --client-ca-file="" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788628 2552 flags.go:64] FLAG: --cloud-provider="external" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788631 2552 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788637 2552 flags.go:64] FLAG: --cluster-domain="" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788640 2552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788643 2552 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788646 2552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788650 2552 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788653 2552 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788656 2552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788659 2552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788662 2552 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788665 2552 flags.go:64] FLAG: --contention-profiling="false" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788668 2552 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788672 2552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788675 2552 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 21:37:14.122768 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788678 2552 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788682 2552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788686 2552 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788689 2552 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788691 2552 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788695 2552 flags.go:64] FLAG: --enable-server="true" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788705 2552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788712 2552 flags.go:64] FLAG: --event-burst="100" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788715 2552 flags.go:64] FLAG: --event-qps="50" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788717 2552 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788721 2552 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788725 2552 flags.go:64] FLAG: --eviction-hard="" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788729 2552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788732 2552 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788734 2552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788738 2552 flags.go:64] FLAG: --eviction-soft="" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788741 2552 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788743 2552 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788746 2552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788749 2552 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788752 2552 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788755 2552 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788758 2552 flags.go:64] FLAG: --feature-gates="" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788762 2552 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788765 2552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 21:37:14.123485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788768 2552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788771 2552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788774 2552 flags.go:64] FLAG: --healthz-port="10248" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788777 2552 flags.go:64] FLAG: --help="false" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788780 2552 flags.go:64] FLAG: --hostname-override="ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788785 2552 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788788 2552 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788791 2552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788794 2552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788797 2552 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788800 2552 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788803 2552 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788806 2552 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788809 2552 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788820 2552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788823 2552 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788826 2552 flags.go:64] FLAG: --kube-reserved="" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788829 2552 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788833 2552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788836 2552 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788839 2552 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788842 2552 flags.go:64] FLAG: --lock-file="" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788844 2552 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788847 2552 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 21:37:14.124130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788850 2552 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788855 2552 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788859 2552 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788861 2552 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788864 2552 flags.go:64] FLAG: --logging-format="text" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788867 2552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788870 2552 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788873 2552 flags.go:64] FLAG: --manifest-url="" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788875 2552 flags.go:64] FLAG: --manifest-url-header="" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788880 2552 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788883 2552 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788887 2552 flags.go:64] FLAG: --max-pods="110" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788890 2552 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788894 2552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788897 2552 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788900 2552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788902 2552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788905 2552 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788908 2552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788916 2552 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788919 2552 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788921 2552 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788924 2552 flags.go:64] FLAG: --pod-cidr="" Jul 14 21:37:14.124834 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788933 2552 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788939 2552 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788941 2552 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788946 2552 flags.go:64] FLAG: --pods-per-core="0" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788949 2552 flags.go:64] FLAG: --port="10250" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788952 2552 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788955 2552 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d5138d299c2d923f" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788958 2552 flags.go:64] FLAG: --qos-reserved="" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788961 2552 flags.go:64] FLAG: --read-only-port="10255" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788964 2552 flags.go:64] FLAG: --register-node="true" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788967 2552 flags.go:64] FLAG: --register-with-taints="" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788971 2552 flags.go:64] FLAG: --registry-burst="10" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788974 2552 flags.go:64] FLAG: --registry-qps="5" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788976 2552 flags.go:64] FLAG: --reserved-cpus="" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788979 2552 flags.go:64] FLAG: --reserved-memory="" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788983 2552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788985 2552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788988 2552 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788991 2552 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788994 2552 flags.go:64] FLAG: --runonce="false" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.788997 2552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789000 2552 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789003 2552 flags.go:64] FLAG: --seccomp-default="false" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789007 2552 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789010 2552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789013 2552 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 21:37:14.125734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789016 2552 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789019 2552 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789022 2552 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789025 2552 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789027 2552 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789030 2552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789033 2552 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789042 2552 flags.go:64] FLAG: --system-cgroups="" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789045 2552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789052 2552 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789055 2552 flags.go:64] FLAG: --tls-cert-file="" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789058 2552 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789064 2552 flags.go:64] FLAG: --tls-min-version="" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789067 2552 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789070 2552 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789072 2552 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789075 2552 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789078 2552 flags.go:64] FLAG: --v="2" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789082 2552 flags.go:64] FLAG: --version="false" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789086 2552 flags.go:64] FLAG: --vmodule="" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789090 2552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.789093 2552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789268 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789273 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 21:37:14.126476 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789276 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789279 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789282 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789285 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789288 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789292 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789294 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789297 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789299 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789302 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789304 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789307 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789309 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789312 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789315 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789317 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789326 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789330 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789332 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 21:37:14.261836 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789335 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789337 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789340 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789343 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789345 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789348 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789351 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789353 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789356 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789359 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789361 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789364 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789366 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789369 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789371 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789374 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789376 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789380 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789385 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 21:37:14.262527 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789388 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789391 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789394 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789397 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789399 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789402 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789404 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789407 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789409 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789412 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789414 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789419 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789421 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789424 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789426 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789430 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789434 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789436 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789439 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 21:37:14.263040 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789442 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789445 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789448 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789451 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789454 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789456 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789459 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789462 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789464 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789467 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789470 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789472 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789476 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789479 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789482 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789484 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789487 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789490 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789492 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789495 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 21:37:14.263591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789498 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789500 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789503 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789505 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789509 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789512 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789515 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789517 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789520 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789522 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789525 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789527 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789530 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789534 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789537 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789540 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789543 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789546 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789549 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 21:37:14.264108 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.789551 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 21:37:14.264645 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.790232 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 21:37:14.264645 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.798020 2552 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 21:37:14.264645 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.798042 2552 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 21:37:14.264645 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.798062 2552 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 21:37:14.264645 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.798069 2552 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 21:37:14.264645 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798150 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 21:37:14.264645 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798156 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 21:37:14.264645 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798159 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 21:37:14.264645 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798163 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 21:37:14.264645 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798165 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 21:37:14.264645 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798168 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 21:37:14.264645 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798171 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 21:37:14.264645 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798173 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 21:37:14.264645 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798176 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 21:37:14.264645 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798178 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798182 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798185 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798203 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798206 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798209 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798212 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798214 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798217 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798220 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798222 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798225 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798227 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798230 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798234 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798238 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798241 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798244 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798247 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798250 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 21:37:14.265113 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798259 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798262 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798264 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798267 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798269 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798272 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798274 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798277 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798279 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798282 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798285 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798287 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798290 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798293 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798295 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798298 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798301 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798304 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798307 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798309 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 21:37:14.265688 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798312 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798314 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798317 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798319 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798322 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798325 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798327 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798330 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798332 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798335 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798338 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798340 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798343 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798351 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798354 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798356 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798360 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798364 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798366 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798369 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 21:37:14.266225 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798371 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798374 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798376 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798379 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798381 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798384 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798386 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798389 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798392 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798395 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798398 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798400 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798403 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798405 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798409 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798412 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798415 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798421 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798424 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 21:37:14.266787 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798427 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 21:37:14.267303 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798429 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 21:37:14.267303 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798432 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 21:37:14.267303 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798435 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 21:37:14.267303 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798437 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 21:37:14.267303 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798440 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 21:37:14.267303 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798442 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 21:37:14.267303 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798450 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 21:37:14.267303 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798453 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 21:37:14.267303 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798456 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 21:37:14.267303 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798458 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 21:37:14.267303 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.798463 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 21:37:14.267303 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798615 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 21:37:14.267303 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798620 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 21:37:14.267303 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798623 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798627 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798631 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798634 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798637 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798640 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798644 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798647 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798650 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798653 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798656 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798659 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798661 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798664 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798666 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798669 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798671 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798675 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798677 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798680 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 21:37:14.267724 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798683 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798685 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798688 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798690 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798693 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798695 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798704 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798707 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798709 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798712 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798714 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798717 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798720 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798722 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798725 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798728 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798730 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798732 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798738 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 21:37:14.268246 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798741 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798744 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798746 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798749 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798751 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798754 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798756 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798759 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798763 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798766 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798771 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798773 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798776 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798779 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798787 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798791 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798793 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798796 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798798 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798801 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 21:37:14.268825 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798809 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798812 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798815 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798817 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798820 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798822 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798825 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798827 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798830 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798832 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798835 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798839 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798841 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798844 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798846 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798848 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798851 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798854 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798856 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798859 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 21:37:14.269369 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798861 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 21:37:14.269935 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798863 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 21:37:14.269935 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798867 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 21:37:14.269935 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798870 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 21:37:14.269935 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798872 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 21:37:14.269935 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798874 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 21:37:14.269935 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798877 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 21:37:14.269935 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798879 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 21:37:14.269935 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798882 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 21:37:14.269935 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798884 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 21:37:14.269935 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798887 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 21:37:14.269935 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798889 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 21:37:14.269935 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798892 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 21:37:14.269935 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798901 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 21:37:14.269935 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798904 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 21:37:14.269935 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798907 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 21:37:14.269935 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798909 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 21:37:14.269935 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:13.798912 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 21:37:14.270410 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.798917 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 21:37:14.270410 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.799057 2552 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 21:37:14.270410 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.801555 2552 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 21:37:14.270410 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.803159 2552 server.go:1019] "Starting client certificate rotation" Jul 14 21:37:14.270410 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.803278 2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 21:37:14.270410 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.803323 2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 21:37:14.270410 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.806465 2552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 21:37:14.270410 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.813383 2552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 21:37:14.270410 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.827333 2552 log.go:25] "Validated CRI v1 runtime API" Jul 14 21:37:14.270410 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.829751 2552 log.go:25] "Validated CRI v1 image API" Jul 14 21:37:14.270410 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.830679 2552 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 21:37:14.270410 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.833571 2552 fs.go:135] Filesystem UUIDs: map[44e61d06-8bfd-4a75-bc3d-32cbff3c037f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cac93cd6-1118-44b2-8434-d49d0df30fbf:/dev/nvme0n1p3] Jul 14 21:37:14.270787 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.833588 2552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 21:37:14.270859 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.841813 2552 manager.go:216] Machine: {Timestamp:2026-07-14 21:37:13.840461713 +0000 UTC m=+0.379167378 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3065527 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e332b68042b3c3082b3eea2ad0e4d SystemUUID:ec2e332b-6804-2b3c-3082-b3eea2ad0e4d BootID:34c98588-e854-4f40-b5a1-afb1d4423d6e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:97:7f:8b:eb:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:97:7f:8b:eb:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:d8:38:50:d4:20 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 21:37:14.270859 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.841917 2552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 21:37:14.270859 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.842057 2552 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 21:37:14.270859 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.845520 2552 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 21:37:14.270859 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.845631 2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 21:37:14.270859 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.845563 2552 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-223.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 21:37:14.270859 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.846517 2552 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.846531 2552 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.846538 2552 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.847315 2552 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.849021 2552 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.852392 2552 state_mem.go:36] "Initialized new in-memory state store" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.852515 2552 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.852545 2552 kubelet.go:486] "Attempting to sync node with API server" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.852556 2552 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.852569 2552 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.852579 2552 kubelet.go:398] "Adding apiserver pod source" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.852587 2552 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.852953 2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.852972 2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.855429 2552 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.857728 2552 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.857756 2552 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.857914 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.857926 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.857933 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.857938 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.857953 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.857961 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.858607 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.858615 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.858622 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.858627 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.858637 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.858645 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.859434 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.859442 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.862856 2552 server.go:1268] "Started kubelet" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.862936 2552 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.862961 2552 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.863073 2552 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.868380 2552 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.869209 2552 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-223.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:13.869308 2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:13.869347 2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-223.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 21:37:14.271141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.870639 2552 server.go:310] "Adding debug handlers to kubelet server" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:13.873351 2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-223.ec2.internal.18c245b22796d0bb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-223.ec2.internal,UID:ip-10-0-130-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-223.ec2.internal,},FirstTimestamp:2026-07-14 21:37:13.862828219 +0000 UTC m=+0.401533881,LastTimestamp:2026-07-14 21:37:13.862828219 +0000 UTC m=+0.401533881,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-223.ec2.internal,}" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:13.875543 2552 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.878566 2552 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.878578 2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:13.878658 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.879276 2552 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.879290 2552 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.879373 2552 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.879379 2552 factory.go:55] Registering systemd factory Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.879410 2552 factory.go:223] Registration of the systemd container factory successfully Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.879381 2552 reconstruct.go:97] "Volume reconstruction finished" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.879463 2552 reconciler.go:29] "Reconciler: start to sync state" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.879623 2552 factory.go:153] Registering CRI-O factory Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.879635 2552 factory.go:223] Registration of the crio container factory successfully Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.879685 2552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.879704 2552 factory.go:103] Registering Raw factory Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.879714 2552 manager.go:1196] Started watching for new ooms in manager Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.880141 2552 manager.go:349] Starting recovery of all containers Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.886496 2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:13.891366 2552 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-223.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:13.891419 2552 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.891923 2552 manager.go:354] Recovery completed Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.896241 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.897339 2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jcfj4" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.899271 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientMemory" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.899519 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.899530 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientPID" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.901213 2552 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.901228 2552 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.901233 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 21:37:14.469926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.901244 2552 state_mem.go:36] "Initialized new in-memory state store" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:13.901545 2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-223.ec2.internal.18c245b229c35c70 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-223.ec2.internal,UID:ip-10-0-130-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-223.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-223.ec2.internal,},FirstTimestamp:2026-07-14 21:37:13.899302 +0000 UTC m=+0.438007665,LastTimestamp:2026-07-14 21:37:13.899302 +0000 UTC m=+0.438007665,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-223.ec2.internal,}" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.902059 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientMemory" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.902087 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.902101 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientPID" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.902536 2552 policy_none.go:49] "None policy: Start" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.902552 2552 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.902562 2552 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.903666 2552 policy_none.go:47] "Start" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.906995 2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jcfj4" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:13.912403 2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-223.ec2.internal.18c245b229c6c0cb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-223.ec2.internal,UID:ip-10-0-130-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-223.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-223.ec2.internal,},FirstTimestamp:2026-07-14 21:37:13.899524299 +0000 UTC m=+0.438229962,LastTimestamp:2026-07-14 21:37:13.899524299 +0000 UTC m=+0.438229962,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-223.ec2.internal,}" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:13.978906 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.985383 2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.985427 2552 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.985450 2552 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:13.985487 2552 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.988479 2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.989150 2552 manager.go:337] "Starting Device Plugin manager" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:13.989180 2552 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.989211 2552 server.go:87] "Starting device plugin registration server" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.989489 2552 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.989501 2552 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.989597 2552 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.989674 2552 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:13.989693 2552 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:13.990233 2552 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 21:37:14.471794 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:13.990272 2552 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-223.ec2.internal\" not found" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.086155 2552 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal"] Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.086242 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.087855 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientMemory" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.087888 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.087902 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientPID" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.089205 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.089380 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.089407 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.089659 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.089980 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientMemory" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.090004 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.090018 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientPID" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.090025 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientMemory" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.090053 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.090067 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientPID" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.090360 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientMemory" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.090388 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.090400 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientPID" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.090423 2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.091045 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.091071 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.091750 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientMemory" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.091772 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.091790 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientPID" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.100784 2552 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:14.100805 2552 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-223.ec2.internal\": node \"ip-10-0-130-223.ec2.internal\" not found" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:14.156328 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:14.159755 2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-223.ec2.internal\" not found" node="ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:14.165376 2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-223.ec2.internal\" not found" node="ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.473384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.180943 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2a59ded2d15e5a27ff495bdc28701515-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal\" (UID: \"2a59ded2d15e5a27ff495bdc28701515\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.474313 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.180966 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a59ded2d15e5a27ff495bdc28701515-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal\" (UID: \"2a59ded2d15e5a27ff495bdc28701515\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.474313 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.180982 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6347ca5f52a32e5fefe18e2f2e47b4f4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-223.ec2.internal\" (UID: \"6347ca5f52a32e5fefe18e2f2e47b4f4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.474313 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:14.256890 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found" Jul 14 21:37:14.474313 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.281498 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2a59ded2d15e5a27ff495bdc28701515-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal\" (UID: \"2a59ded2d15e5a27ff495bdc28701515\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.474313 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.281525 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a59ded2d15e5a27ff495bdc28701515-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal\" (UID: \"2a59ded2d15e5a27ff495bdc28701515\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.474313 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.281546 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6347ca5f52a32e5fefe18e2f2e47b4f4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-223.ec2.internal\" (UID: \"6347ca5f52a32e5fefe18e2f2e47b4f4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.474313 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.281600 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2a59ded2d15e5a27ff495bdc28701515-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal\" (UID: \"2a59ded2d15e5a27ff495bdc28701515\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.474313 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.281612 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a59ded2d15e5a27ff495bdc28701515-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal\" (UID: \"2a59ded2d15e5a27ff495bdc28701515\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.474313 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.281600 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6347ca5f52a32e5fefe18e2f2e47b4f4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-223.ec2.internal\" (UID: \"6347ca5f52a32e5fefe18e2f2e47b4f4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.474313 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:14.357963 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found" Jul 14 21:37:14.474313 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:14.458495 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found" Jul 14 21:37:14.474313 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.461658 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.474313 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.468319 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal" Jul 14 21:37:14.661313 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:14.559047 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found" Jul 14 21:37:14.661313 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:14.659905 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found" Jul 14 21:37:14.850527 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:14.763352 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found" Jul 14 21:37:14.850527 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.802247 2552 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 21:37:14.850527 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.802385 2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 21:37:15.012835 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:14.864489 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found" Jul 14 21:37:15.012835 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.897007 2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 21:37:15.012835 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.910666 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 21:32:14 +0000 UTC" deadline="2028-04-25 04:20:36.552857032 +0000 UTC" Jul 14 21:37:15.012835 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.910698 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15606h43m21.642162257s" Jul 14 21:37:15.012835 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.919054 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 21:37:15.012835 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.920678 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientMemory" Jul 14 21:37:15.012835 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.920711 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 21:37:15.012835 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:14.920724 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientPID" Jul 14 21:37:15.012835 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:14.965722 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found" Jul 14 21:37:15.191676 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:15.066620 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found" Jul 14 21:37:15.191676 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.098320 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 21:37:15.191676 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:15.168849 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found" Jul 14 21:37:15.367100 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.256287 2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 21:37:15.367100 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:15.269390 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found" Jul 14 21:37:15.513983 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:15.370037 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found" Jul 14 21:37:15.513983 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.454536 2552 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 21:37:15.513983 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.478770 2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal" Jul 14 21:37:15.513983 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.513338 2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 21:37:15.515614 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.515405 2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal" Jul 14 21:37:15.687201 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.535276 2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.855876 2552 apiserver.go:52] "Watching apiserver" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.862439 2552 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.864934 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg","openshift-cluster-node-tuning-operator/tuned-xnl8n","openshift-dns/node-resolver-c4vgt","openshift-image-registry/node-ca-2545q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal","openshift-network-diagnostics/network-check-target-8hxtk","kube-system/konnectivity-agent-9thwv","openshift-multus/multus-additional-cni-plugins-2k4fl","openshift-multus/multus-p4twf","openshift-multus/network-metrics-daemon-czmxj","openshift-network-operator/iptables-alerter-dctmt","openshift-ovn-kubernetes/ovnkube-node-fxphv","kube-system/global-pull-secret-syncer-f2b5s"] Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.871220 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:15.871318 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hxtk" podUID="8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.875682 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.878591 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.878937 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.879780 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.879968 2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.880087 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qjj8n\"" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.880340 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.880514 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.880705 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.881105 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c4vgt" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.881254 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.883329 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2545q" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.883700 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9djtc\"" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.884053 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.884092 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.885625 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:15.885727 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2b5s" podUID="d0336426-165c-4987-beaa-f74fe715d480" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.886657 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wvptx\"" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.887076 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.887434 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 21:37:15.922943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.888074 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 21:37:15.924670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.888336 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 21:37:15.924670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.888536 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 21:37:15.924670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.888558 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:15.924670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.888767 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fxc85\"" Jul 14 21:37:15.924670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.890732 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9thwv" Jul 14 21:37:15.924670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.891849 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lffmx\" (UniqueName: \"kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx\") pod \"network-check-target-8hxtk\" (UID: \"8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692\") " pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:15.924670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.891889 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-log-socket\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.924670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.891945 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-run-ovn-kubernetes\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.924670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.891975 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-systemd-units\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.924670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892040 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24ckj\" (UniqueName: \"kubernetes.io/projected/b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10-kube-api-access-24ckj\") pod \"node-ca-2545q\" (UID: \"b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10\") " pod="openshift-image-registry/node-ca-2545q" Jul 14 21:37:15.924670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892076 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-cni-netd\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.924670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892111 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/239cdaed-5bca-4561-80be-9bac280ce4b3-tmp\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:15.924670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892148 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba716d4b-af3d-44a4-a7d0-d134e8c805f3-hosts-file\") pod \"node-resolver-c4vgt\" (UID: \"ba716d4b-af3d-44a4-a7d0-d134e8c805f3\") " pod="openshift-dns/node-resolver-c4vgt" Jul 14 21:37:15.924670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892180 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9cc0cd22-ce64-4aaf-8af0-2618da80d640-env-overrides\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.924670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892217 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9cc0cd22-ce64-4aaf-8af0-2618da80d640-ovn-node-metrics-cert\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.924670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892253 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-sysctl-d\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:15.925998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892325 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-systemd\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:15.925998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892380 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-run\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:15.925998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892413 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-run-ovn\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.925998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892448 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba716d4b-af3d-44a4-a7d0-d134e8c805f3-tmp-dir\") pod \"node-resolver-c4vgt\" (UID: \"ba716d4b-af3d-44a4-a7d0-d134e8c805f3\") " pod="openshift-dns/node-resolver-c4vgt" Jul 14 21:37:15.925998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892494 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gjzj\" (UniqueName: \"kubernetes.io/projected/ba716d4b-af3d-44a4-a7d0-d134e8c805f3-kube-api-access-9gjzj\") pod \"node-resolver-c4vgt\" (UID: \"ba716d4b-af3d-44a4-a7d0-d134e8c805f3\") " pod="openshift-dns/node-resolver-c4vgt" Jul 14 21:37:15.925998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892528 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10-serviceca\") pod \"node-ca-2545q\" (UID: \"b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10\") " pod="openshift-image-registry/node-ca-2545q" Jul 14 21:37:15.925998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892570 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-modprobe-d\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:15.925998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892606 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-sysconfig\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:15.925998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892632 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-sysctl-conf\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:15.925998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892651 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-sys\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:15.925998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892671 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-etc-openvswitch\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.925998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892689 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-node-log\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.925998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892708 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9cc0cd22-ce64-4aaf-8af0-2618da80d640-ovnkube-script-lib\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.925998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892726 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-var-lib-kubelet\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:15.927413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892748 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-tuned\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:15.927413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892768 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcv8g\" (UniqueName: \"kubernetes.io/projected/239cdaed-5bca-4561-80be-9bac280ce4b3-kube-api-access-vcv8g\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:15.927413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892786 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10-host\") pod \"node-ca-2545q\" (UID: \"b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10\") " pod="openshift-image-registry/node-ca-2545q" Jul 14 21:37:15.927413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892814 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-run-systemd\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.927413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892837 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-run-openvswitch\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.927413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892863 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-cni-bin\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.927413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892893 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcnwn\" (UniqueName: \"kubernetes.io/projected/9cc0cd22-ce64-4aaf-8af0-2618da80d640-kube-api-access-xcnwn\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.927413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892912 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.927413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892933 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-kubernetes\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:15.927413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892954 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9cc0cd22-ce64-4aaf-8af0-2618da80d640-ovnkube-config\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.927413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892972 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-slash\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.927413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.892990 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-run-netns\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.927413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.893016 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-kubelet\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.927413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.893031 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-var-lib-openvswitch\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.893049 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-lib-modules\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.893069 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-host\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.893770 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.893833 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rg4fm\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.894070 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.894199 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.894067 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.894320 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.894375 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.895378 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.895446 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.895455 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-57frc\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.898725 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p4twf" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.900978 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.901157 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.901243 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m9g66\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.901522 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:15.901601 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czmxj" podUID="fab63e1e-b743-4ed6-bbbd-6686855f1a43" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.903129 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.903391 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-b9nqd\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.903904 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dctmt" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.904369 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.907310 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.907464 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.907753 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.907845 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m8blk\"" Jul 14 21:37:15.929145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.907959 2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 21:37:16.009326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.954742 2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rvtzh" Jul 14 21:37:16.009326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.960244 2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 21:37:16.009326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.970966 2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rvtzh" Jul 14 21:37:16.009326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.980845 2552 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 21:37:16.009326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993058 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal" event={"ID":"2a59ded2d15e5a27ff495bdc28701515","Type":"ContainerStarted","Data":"7dccdd0799ad59880ed63f691131d4f2f73bc6203dbeadb0ad427240b3ae9efc"} Jul 14 21:37:16.009326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993252 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-sysconfig\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.009326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993289 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-tuned\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.009326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993322 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-var-lib-cni-bin\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.009326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993352 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a255c4c-c31b-4a1b-bd13-263359830473-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.009326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993381 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-multus-cni-dir\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.009326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993408 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzsgr\" (UniqueName: \"kubernetes.io/projected/6d5ffe2e-7323-41c6-a850-570f6a627f16-kube-api-access-gzsgr\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.009326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993436 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a255c4c-c31b-4a1b-bd13-263359830473-os-release\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.009326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993462 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/451155f1-ad03-4be3-b198-895002675826-agent-certs\") pod \"konnectivity-agent-9thwv\" (UID: \"451155f1-ad03-4be3-b198-895002675826\") " pod="kube-system/konnectivity-agent-9thwv" Jul 14 21:37:16.009326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993488 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10-host\") pod \"node-ca-2545q\" (UID: \"b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10\") " pod="openshift-image-registry/node-ca-2545q" Jul 14 21:37:16.009326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993518 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-cni-bin\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.009326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993542 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcnwn\" (UniqueName: \"kubernetes.io/projected/9cc0cd22-ce64-4aaf-8af0-2618da80d640-kube-api-access-xcnwn\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.010539 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993569 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t25r2\" (UniqueName: \"kubernetes.io/projected/bbd4582f-0e8a-4b90-a668-7f5260893841-kube-api-access-t25r2\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.010539 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993590 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.010539 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993607 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-hostroot\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.010539 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993644 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6b4r\" (UniqueName: \"kubernetes.io/projected/2a255c4c-c31b-4a1b-bd13-263359830473-kube-api-access-w6b4r\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.010539 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993670 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d0336426-165c-4987-beaa-f74fe715d480-dbus\") pod \"global-pull-secret-syncer-f2b5s\" (UID: \"d0336426-165c-4987-beaa-f74fe715d480\") " pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:16.010539 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993697 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9cc0cd22-ce64-4aaf-8af0-2618da80d640-ovnkube-config\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.010539 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993734 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-var-lib-openvswitch\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.010539 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993786 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-host\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.010539 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993777 2552 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 21:37:16.010539 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993821 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a255c4c-c31b-4a1b-bd13-263359830473-cnibin\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.010539 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993852 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lffmx\" (UniqueName: \"kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx\") pod \"network-check-target-8hxtk\" (UID: \"8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692\") " pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:16.010539 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993879 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-run-ovn-kubernetes\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.010539 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.993923 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-run-k8s-cni-cncf-io\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.010539 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.994263 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10-host\") pod \"node-ca-2545q\" (UID: \"b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10\") " pod="openshift-image-registry/node-ca-2545q" Jul 14 21:37:16.011399 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.994306 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-cni-bin\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.011399 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.994331 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwg6r\" (UniqueName: \"kubernetes.io/projected/fab63e1e-b743-4ed6-bbbd-6686855f1a43-kube-api-access-vwg6r\") pod \"network-metrics-daemon-czmxj\" (UID: \"fab63e1e-b743-4ed6-bbbd-6686855f1a43\") " pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:16.011399 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.994358 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-host\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.011399 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.994370 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/239cdaed-5bca-4561-80be-9bac280ce4b3-tmp\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.011399 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.994406 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-device-dir\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.011399 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.994405 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-sysconfig\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.011399 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.994486 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-sys-fs\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.011399 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.994492 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-run-ovn-kubernetes\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.011399 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.994710 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-var-lib-openvswitch\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.011399 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.994731 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a255c4c-c31b-4a1b-bd13-263359830473-cni-binary-copy\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.011399 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.995454 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9cc0cd22-ce64-4aaf-8af0-2618da80d640-ovnkube-config\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.011399 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.996874 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9njnx\" (UniqueName: \"kubernetes.io/projected/9ff48b38-946a-48b0-bf80-d46cee550a8b-kube-api-access-9njnx\") pod \"iptables-alerter-dctmt\" (UID: \"9ff48b38-946a-48b0-bf80-d46cee550a8b\") " pod="openshift-network-operator/iptables-alerter-dctmt" Jul 14 21:37:16.011399 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.996913 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-sysctl-d\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.011399 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.996939 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-systemd\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.013207 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.996967 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d0336426-165c-4987-beaa-f74fe715d480-kubelet-config\") pod \"global-pull-secret-syncer-f2b5s\" (UID: \"d0336426-165c-4987-beaa-f74fe715d480\") " pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:16.013207 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997035 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-socket-dir\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.013207 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997083 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-sysctl-d\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.013207 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997079 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-registration-dir\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.013207 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997118 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-etc-selinux\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.013207 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997115 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-systemd\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.013207 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997149 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-multus-conf-dir\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.013207 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997177 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d5ffe2e-7323-41c6-a850-570f6a627f16-multus-daemon-config\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.013207 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997205 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjzj\" (UniqueName: \"kubernetes.io/projected/ba716d4b-af3d-44a4-a7d0-d134e8c805f3-kube-api-access-9gjzj\") pod \"node-resolver-c4vgt\" (UID: \"ba716d4b-af3d-44a4-a7d0-d134e8c805f3\") " pod="openshift-dns/node-resolver-c4vgt" Jul 14 21:37:16.013207 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997242 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10-serviceca\") pod \"node-ca-2545q\" (UID: \"b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10\") " pod="openshift-image-registry/node-ca-2545q" Jul 14 21:37:16.013207 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997266 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-sysctl-conf\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.013207 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997288 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-sys\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.013207 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997316 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-var-lib-kubelet\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.013207 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997355 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcv8g\" (UniqueName: \"kubernetes.io/projected/239cdaed-5bca-4561-80be-9bac280ce4b3-kube-api-access-vcv8g\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.014564 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997380 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-var-lib-cni-multus\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.014564 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997404 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a255c4c-c31b-4a1b-bd13-263359830473-system-cni-dir\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.014564 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997430 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-etc-openvswitch\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.014564 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997452 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-node-log\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.014564 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997475 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9cc0cd22-ce64-4aaf-8af0-2618da80d640-ovnkube-script-lib\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.014564 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997498 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-os-release\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.014564 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997522 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a255c4c-c31b-4a1b-bd13-263359830473-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.014564 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997550 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/451155f1-ad03-4be3-b198-895002675826-konnectivity-ca\") pod \"konnectivity-agent-9thwv\" (UID: \"451155f1-ad03-4be3-b198-895002675826\") " pod="kube-system/konnectivity-agent-9thwv" Jul 14 21:37:16.014564 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997574 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9ff48b38-946a-48b0-bf80-d46cee550a8b-iptables-alerter-script\") pod \"iptables-alerter-dctmt\" (UID: \"9ff48b38-946a-48b0-bf80-d46cee550a8b\") " pod="openshift-network-operator/iptables-alerter-dctmt" Jul 14 21:37:16.014564 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997606 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-run-systemd\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.014564 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997622 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-sysctl-conf\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.014564 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997629 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-run-openvswitch\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.014564 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997663 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-run-netns\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.014564 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997688 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-var-lib-kubelet\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.016647 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997686 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-run-multus-certs\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.016647 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997722 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a255c4c-c31b-4a1b-bd13-263359830473-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.016647 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997745 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ff48b38-946a-48b0-bf80-d46cee550a8b-host-slash\") pod \"iptables-alerter-dctmt\" (UID: \"9ff48b38-946a-48b0-bf80-d46cee550a8b\") " pod="openshift-network-operator/iptables-alerter-dctmt" Jul 14 21:37:16.016647 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997781 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.016647 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997827 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-kubernetes\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.016647 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997836 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10-serviceca\") pod \"node-ca-2545q\" (UID: \"b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10\") " pod="openshift-image-registry/node-ca-2545q" Jul 14 21:37:16.016647 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997664 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-sys\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.016647 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997851 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-cnibin\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.016647 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997889 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret\") pod \"global-pull-secret-syncer-f2b5s\" (UID: \"d0336426-165c-4987-beaa-f74fe715d480\") " pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:16.016647 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997898 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-run-systemd\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.016647 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997911 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-slash\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.016647 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997931 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-run-netns\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.016647 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997950 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-kubelet\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.016647 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997949 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-kubernetes\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.017571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997980 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-kubelet\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.017571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.997999 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-lib-modules\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.017571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998001 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-etc-openvswitch\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.017571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998029 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-run-openvswitch\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.017571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998071 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-log-socket\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.017571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998106 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-system-cni-dir\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.017571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998137 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-systemd-units\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.017571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998131 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.017571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998167 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24ckj\" (UniqueName: \"kubernetes.io/projected/b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10-kube-api-access-24ckj\") pod \"node-ca-2545q\" (UID: \"b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10\") " pod="openshift-image-registry/node-ca-2545q" Jul 14 21:37:16.017571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998113 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-lib-modules\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.017571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998178 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-slash\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.017571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998215 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-run-netns\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.017571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998231 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-systemd-units\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.017571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998242 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-node-log\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.017571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998287 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-log-socket\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.020566 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998285 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-cni-netd\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.020566 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998318 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-host-cni-netd\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.020566 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998323 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-etc-kubernetes\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.020566 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998342 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs\") pod \"network-metrics-daemon-czmxj\" (UID: \"fab63e1e-b743-4ed6-bbbd-6686855f1a43\") " pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:16.020566 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998371 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9cc0cd22-ce64-4aaf-8af0-2618da80d640-ovnkube-script-lib\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.020566 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998456 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba716d4b-af3d-44a4-a7d0-d134e8c805f3-hosts-file\") pod \"node-resolver-c4vgt\" (UID: \"ba716d4b-af3d-44a4-a7d0-d134e8c805f3\") " pod="openshift-dns/node-resolver-c4vgt" Jul 14 21:37:16.020566 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998493 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9cc0cd22-ce64-4aaf-8af0-2618da80d640-env-overrides\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.020566 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998526 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9cc0cd22-ce64-4aaf-8af0-2618da80d640-ovn-node-metrics-cert\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.020566 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998555 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba716d4b-af3d-44a4-a7d0-d134e8c805f3-hosts-file\") pod \"node-resolver-c4vgt\" (UID: \"ba716d4b-af3d-44a4-a7d0-d134e8c805f3\") " pod="openshift-dns/node-resolver-c4vgt" Jul 14 21:37:16.020566 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998596 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-run\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.020566 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998623 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d5ffe2e-7323-41c6-a850-570f6a627f16-cni-binary-copy\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.020566 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.998767 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-run\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.020566 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.999017 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9cc0cd22-ce64-4aaf-8af0-2618da80d640-env-overrides\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.020566 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.999066 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-multus-socket-dir-parent\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.020566 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.999096 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-var-lib-kubelet\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.021924 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.999159 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-run-ovn\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.021924 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.999215 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba716d4b-af3d-44a4-a7d0-d134e8c805f3-tmp-dir\") pod \"node-resolver-c4vgt\" (UID: \"ba716d4b-af3d-44a4-a7d0-d134e8c805f3\") " pod="openshift-dns/node-resolver-c4vgt" Jul 14 21:37:16.021924 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.999256 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-modprobe-d\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.021924 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.999422 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-modprobe-d\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.021924 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.999460 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9cc0cd22-ce64-4aaf-8af0-2618da80d640-run-ovn\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.021924 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:15.999753 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba716d4b-af3d-44a4-a7d0-d134e8c805f3-tmp-dir\") pod \"node-resolver-c4vgt\" (UID: \"ba716d4b-af3d-44a4-a7d0-d134e8c805f3\") " pod="openshift-dns/node-resolver-c4vgt" Jul 14 21:37:16.021924 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.003938 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/239cdaed-5bca-4561-80be-9bac280ce4b3-etc-tuned\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.021924 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.006496 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/239cdaed-5bca-4561-80be-9bac280ce4b3-tmp\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.021924 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.006690 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9cc0cd22-ce64-4aaf-8af0-2618da80d640-ovn-node-metrics-cert\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.021924 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:16.008945 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 21:37:16.021924 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:16.008967 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 21:37:16.021924 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:16.008982 2552 projected.go:196] Error preparing data for projected volume kube-api-access-lffmx for pod openshift-network-diagnostics/network-check-target-8hxtk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:37:16.021924 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:16.009062 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx podName:8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:16.50902588 +0000 UTC m=+3.047731547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lffmx" (UniqueName: "kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx") pod "network-check-target-8hxtk" (UID: "8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:37:16.021924 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.009661 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcnwn\" (UniqueName: \"kubernetes.io/projected/9cc0cd22-ce64-4aaf-8af0-2618da80d640-kube-api-access-xcnwn\") pod \"ovnkube-node-fxphv\" (UID: \"9cc0cd22-ce64-4aaf-8af0-2618da80d640\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.021924 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.010800 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal" event={"ID":"6347ca5f52a32e5fefe18e2f2e47b4f4","Type":"ContainerStarted","Data":"da116a8cb59f56da2b6963cef13bfacf6e56dd2cf69b760eb9c8dc74eb0dcd24"} Jul 14 21:37:16.032917 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.028943 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24ckj\" (UniqueName: \"kubernetes.io/projected/b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10-kube-api-access-24ckj\") pod \"node-ca-2545q\" (UID: \"b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10\") " pod="openshift-image-registry/node-ca-2545q" Jul 14 21:37:16.032917 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.029692 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcv8g\" (UniqueName: \"kubernetes.io/projected/239cdaed-5bca-4561-80be-9bac280ce4b3-kube-api-access-vcv8g\") pod \"tuned-xnl8n\" (UID: \"239cdaed-5bca-4561-80be-9bac280ce4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.036899 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.036582 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gjzj\" (UniqueName: \"kubernetes.io/projected/ba716d4b-af3d-44a4-a7d0-d134e8c805f3-kube-api-access-9gjzj\") pod \"node-resolver-c4vgt\" (UID: \"ba716d4b-af3d-44a4-a7d0-d134e8c805f3\") " pod="openshift-dns/node-resolver-c4vgt" Jul 14 21:37:16.119263 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100067 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-run-multus-certs\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.119263 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100109 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a255c4c-c31b-4a1b-bd13-263359830473-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.119263 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100132 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ff48b38-946a-48b0-bf80-d46cee550a8b-host-slash\") pod \"iptables-alerter-dctmt\" (UID: \"9ff48b38-946a-48b0-bf80-d46cee550a8b\") " pod="openshift-network-operator/iptables-alerter-dctmt" Jul 14 21:37:16.119263 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100183 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-run-multus-certs\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.119263 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100202 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ff48b38-946a-48b0-bf80-d46cee550a8b-host-slash\") pod \"iptables-alerter-dctmt\" (UID: \"9ff48b38-946a-48b0-bf80-d46cee550a8b\") " pod="openshift-network-operator/iptables-alerter-dctmt" Jul 14 21:37:16.119263 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100238 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-cnibin\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.119263 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100279 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret\") pod \"global-pull-secret-syncer-f2b5s\" (UID: \"d0336426-165c-4987-beaa-f74fe715d480\") " pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:16.119263 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100308 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-system-cni-dir\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.119263 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100332 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-etc-kubernetes\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.119263 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100353 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs\") pod \"network-metrics-daemon-czmxj\" (UID: \"fab63e1e-b743-4ed6-bbbd-6686855f1a43\") " pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:16.119263 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100378 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d5ffe2e-7323-41c6-a850-570f6a627f16-cni-binary-copy\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.119263 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100396 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-system-cni-dir\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.119263 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100400 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-multus-socket-dir-parent\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.119263 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100434 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-var-lib-kubelet\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.119263 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100454 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-multus-socket-dir-parent\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.120504 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100465 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-var-lib-cni-bin\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.120504 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:16.100480 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 21:37:16.120504 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100492 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a255c4c-c31b-4a1b-bd13-263359830473-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.120504 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100498 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-var-lib-cni-bin\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.120504 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100517 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-multus-cni-dir\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.120504 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100533 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-var-lib-kubelet\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.120504 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100539 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzsgr\" (UniqueName: \"kubernetes.io/projected/6d5ffe2e-7323-41c6-a850-570f6a627f16-kube-api-access-gzsgr\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.120504 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100350 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-cnibin\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.120504 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:16.100565 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret podName:d0336426-165c-4987-beaa-f74fe715d480 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:16.600540141 +0000 UTC m=+3.139245803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret") pod "global-pull-secret-syncer-f2b5s" (UID: "d0336426-165c-4987-beaa-f74fe715d480") : object "kube-system"/"original-pull-secret" not registered Jul 14 21:37:16.120504 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100581 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-etc-kubernetes\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.120504 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:16.100655 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:37:16.120504 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:16.100747 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs podName:fab63e1e-b743-4ed6-bbbd-6686855f1a43 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:16.60073106 +0000 UTC m=+3.139436710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs") pod "network-metrics-daemon-czmxj" (UID: "fab63e1e-b743-4ed6-bbbd-6686855f1a43") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:37:16.120504 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100774 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a255c4c-c31b-4a1b-bd13-263359830473-os-release\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.120504 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100810 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/451155f1-ad03-4be3-b198-895002675826-agent-certs\") pod \"konnectivity-agent-9thwv\" (UID: \"451155f1-ad03-4be3-b198-895002675826\") " pod="kube-system/konnectivity-agent-9thwv" Jul 14 21:37:16.121336 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100848 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t25r2\" (UniqueName: \"kubernetes.io/projected/bbd4582f-0e8a-4b90-a668-7f5260893841-kube-api-access-t25r2\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.121336 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100852 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-multus-cni-dir\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.121336 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100883 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.121336 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100916 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-hostroot\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.121336 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100923 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a255c4c-c31b-4a1b-bd13-263359830473-os-release\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.121336 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100947 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6b4r\" (UniqueName: \"kubernetes.io/projected/2a255c4c-c31b-4a1b-bd13-263359830473-kube-api-access-w6b4r\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.121336 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.100973 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d0336426-165c-4987-beaa-f74fe715d480-dbus\") pod \"global-pull-secret-syncer-f2b5s\" (UID: \"d0336426-165c-4987-beaa-f74fe715d480\") " pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:16.121336 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101006 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a255c4c-c31b-4a1b-bd13-263359830473-cnibin\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.121336 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101053 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-run-k8s-cni-cncf-io\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.121336 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101080 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-hostroot\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.121336 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101141 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwg6r\" (UniqueName: \"kubernetes.io/projected/fab63e1e-b743-4ed6-bbbd-6686855f1a43-kube-api-access-vwg6r\") pod \"network-metrics-daemon-czmxj\" (UID: \"fab63e1e-b743-4ed6-bbbd-6686855f1a43\") " pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:16.121336 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101172 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-device-dir\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.121336 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101218 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-sys-fs\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.121336 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101247 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a255c4c-c31b-4a1b-bd13-263359830473-cni-binary-copy\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.122519 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101279 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njnx\" (UniqueName: \"kubernetes.io/projected/9ff48b38-946a-48b0-bf80-d46cee550a8b-kube-api-access-9njnx\") pod \"iptables-alerter-dctmt\" (UID: \"9ff48b38-946a-48b0-bf80-d46cee550a8b\") " pod="openshift-network-operator/iptables-alerter-dctmt" Jul 14 21:37:16.122519 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101309 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d0336426-165c-4987-beaa-f74fe715d480-kubelet-config\") pod \"global-pull-secret-syncer-f2b5s\" (UID: \"d0336426-165c-4987-beaa-f74fe715d480\") " pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:16.122519 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101339 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-socket-dir\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.122519 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101374 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-registration-dir\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.122519 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101398 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-etc-selinux\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.122519 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101428 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-multus-conf-dir\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.122519 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101463 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d5ffe2e-7323-41c6-a850-570f6a627f16-multus-daemon-config\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.122519 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101506 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-var-lib-cni-multus\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.122519 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101538 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a255c4c-c31b-4a1b-bd13-263359830473-system-cni-dir\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.122519 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101573 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-os-release\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.122519 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101599 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a255c4c-c31b-4a1b-bd13-263359830473-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.122519 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101624 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/451155f1-ad03-4be3-b198-895002675826-konnectivity-ca\") pod \"konnectivity-agent-9thwv\" (UID: \"451155f1-ad03-4be3-b198-895002675826\") " pod="kube-system/konnectivity-agent-9thwv" Jul 14 21:37:16.122519 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101629 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a255c4c-c31b-4a1b-bd13-263359830473-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.122519 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101654 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9ff48b38-946a-48b0-bf80-d46cee550a8b-iptables-alerter-script\") pod \"iptables-alerter-dctmt\" (UID: \"9ff48b38-946a-48b0-bf80-d46cee550a8b\") " pod="openshift-network-operator/iptables-alerter-dctmt" Jul 14 21:37:16.123315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101690 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-run-netns\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.123315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101701 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.123315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101636 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d5ffe2e-7323-41c6-a850-570f6a627f16-cni-binary-copy\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.123315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101836 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-os-release\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.123315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101841 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-etc-selinux\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.123315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101876 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-multus-conf-dir\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.123315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101903 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a255c4c-c31b-4a1b-bd13-263359830473-cnibin\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.123315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101940 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-run-k8s-cni-cncf-io\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.123315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101993 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d0336426-165c-4987-beaa-f74fe715d480-dbus\") pod \"global-pull-secret-syncer-f2b5s\" (UID: \"d0336426-165c-4987-beaa-f74fe715d480\") " pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:16.123315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.102140 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-device-dir\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.123315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.102185 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a255c4c-c31b-4a1b-bd13-263359830473-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.123315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.102218 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-sys-fs\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.123315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.102345 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d5ffe2e-7323-41c6-a850-570f6a627f16-multus-daemon-config\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.123315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.101693 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d0336426-165c-4987-beaa-f74fe715d480-kubelet-config\") pod \"global-pull-secret-syncer-f2b5s\" (UID: \"d0336426-165c-4987-beaa-f74fe715d480\") " pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:16.123315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.102457 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-var-lib-cni-multus\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.124492 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.102494 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a255c4c-c31b-4a1b-bd13-263359830473-system-cni-dir\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.124492 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.102528 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d5ffe2e-7323-41c6-a850-570f6a627f16-host-run-netns\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.124492 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.102735 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/451155f1-ad03-4be3-b198-895002675826-konnectivity-ca\") pod \"konnectivity-agent-9thwv\" (UID: \"451155f1-ad03-4be3-b198-895002675826\") " pod="kube-system/konnectivity-agent-9thwv" Jul 14 21:37:16.124492 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.102793 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-registration-dir\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.124492 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.102865 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9ff48b38-946a-48b0-bf80-d46cee550a8b-iptables-alerter-script\") pod \"iptables-alerter-dctmt\" (UID: \"9ff48b38-946a-48b0-bf80-d46cee550a8b\") " pod="openshift-network-operator/iptables-alerter-dctmt" Jul 14 21:37:16.124492 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.102875 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bbd4582f-0e8a-4b90-a668-7f5260893841-socket-dir\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.124492 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.102953 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a255c4c-c31b-4a1b-bd13-263359830473-cni-binary-copy\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.124492 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.103395 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a255c4c-c31b-4a1b-bd13-263359830473-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.124492 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.107097 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/451155f1-ad03-4be3-b198-895002675826-agent-certs\") pod \"konnectivity-agent-9thwv\" (UID: \"451155f1-ad03-4be3-b198-895002675826\") " pod="kube-system/konnectivity-agent-9thwv" Jul 14 21:37:16.124492 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.113634 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwg6r\" (UniqueName: \"kubernetes.io/projected/fab63e1e-b743-4ed6-bbbd-6686855f1a43-kube-api-access-vwg6r\") pod \"network-metrics-daemon-czmxj\" (UID: \"fab63e1e-b743-4ed6-bbbd-6686855f1a43\") " pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:16.124492 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.113693 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t25r2\" (UniqueName: \"kubernetes.io/projected/bbd4582f-0e8a-4b90-a668-7f5260893841-kube-api-access-t25r2\") pod \"aws-ebs-csi-driver-node-6srfg\" (UID: \"bbd4582f-0e8a-4b90-a668-7f5260893841\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.124492 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.115270 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzsgr\" (UniqueName: \"kubernetes.io/projected/6d5ffe2e-7323-41c6-a850-570f6a627f16-kube-api-access-gzsgr\") pod \"multus-p4twf\" (UID: \"6d5ffe2e-7323-41c6-a850-570f6a627f16\") " pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.124492 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.116823 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6b4r\" (UniqueName: \"kubernetes.io/projected/2a255c4c-c31b-4a1b-bd13-263359830473-kube-api-access-w6b4r\") pod \"multus-additional-cni-plugins-2k4fl\" (UID: \"2a255c4c-c31b-4a1b-bd13-263359830473\") " pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.124492 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.117244 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njnx\" (UniqueName: \"kubernetes.io/projected/9ff48b38-946a-48b0-bf80-d46cee550a8b-kube-api-access-9njnx\") pod \"iptables-alerter-dctmt\" (UID: \"9ff48b38-946a-48b0-bf80-d46cee550a8b\") " pod="openshift-network-operator/iptables-alerter-dctmt" Jul 14 21:37:16.229806 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.194430 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:16.229806 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.211646 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2545q" Jul 14 21:37:16.229806 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.217401 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" Jul 14 21:37:16.229806 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:16.219513 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc0cd22_ce64_4aaf_8af0_2618da80d640.slice/crio-9bdeb0d12cac33598806bcf2008b83200c40c5e8448a3e2722a67ab417371c42 WatchSource:0}: Error finding container 9bdeb0d12cac33598806bcf2008b83200c40c5e8448a3e2722a67ab417371c42: Status 404 returned error can't find the container with id 9bdeb0d12cac33598806bcf2008b83200c40c5e8448a3e2722a67ab417371c42 Jul 14 21:37:16.229806 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:16.222826 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59fc23b_4e0f_4e2a_8cc7_2bd145c4ab10.slice/crio-695e2af91620f2eb90664eafab6df02806e853c9c316c218ecc5fc2c1a7a126f WatchSource:0}: Error finding container 695e2af91620f2eb90664eafab6df02806e853c9c316c218ecc5fc2c1a7a126f: Status 404 returned error can't find the container with id 695e2af91620f2eb90664eafab6df02806e853c9c316c218ecc5fc2c1a7a126f Jul 14 21:37:16.229806 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.226914 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c4vgt" Jul 14 21:37:16.344016 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.235418 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2k4fl" Jul 14 21:37:16.344016 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:16.239468 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239cdaed_5bca_4561_80be_9bac280ce4b3.slice/crio-de2dcad4e9ad5c733d58d679fb645ae3221b43cf21703658b7c10e6755a4b60f WatchSource:0}: Error finding container de2dcad4e9ad5c733d58d679fb645ae3221b43cf21703658b7c10e6755a4b60f: Status 404 returned error can't find the container with id de2dcad4e9ad5c733d58d679fb645ae3221b43cf21703658b7c10e6755a4b60f Jul 14 21:37:16.344016 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.243159 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9thwv" Jul 14 21:37:16.344016 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.249508 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" Jul 14 21:37:16.344016 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.256344 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p4twf" Jul 14 21:37:16.344016 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:16.264442 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba716d4b_af3d_44a4_a7d0_d134e8c805f3.slice/crio-5478904f8f649d1b629ceb6b492764c769901c46ab861bf6d56e1681a51c06c6 WatchSource:0}: Error finding container 5478904f8f649d1b629ceb6b492764c769901c46ab861bf6d56e1681a51c06c6: Status 404 returned error can't find the container with id 5478904f8f649d1b629ceb6b492764c769901c46ab861bf6d56e1681a51c06c6 Jul 14 21:37:16.344016 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:16.267956 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a255c4c_c31b_4a1b_bd13_263359830473.slice/crio-1c76ae239571d97cc745d2857bb49d419df96cd92e2e82fe65f2b9e1a4711b39 WatchSource:0}: Error finding container 1c76ae239571d97cc745d2857bb49d419df96cd92e2e82fe65f2b9e1a4711b39: Status 404 returned error can't find the container with id 1c76ae239571d97cc745d2857bb49d419df96cd92e2e82fe65f2b9e1a4711b39 Jul 14 21:37:16.344016 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:16.278854 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5ffe2e_7323_41c6_a850_570f6a627f16.slice/crio-a642e4639f062c0537f5e2a91b0edb2c1c4f1589c604ce1ba60d2a7dc2784d44 WatchSource:0}: Error finding container a642e4639f062c0537f5e2a91b0edb2c1c4f1589c604ce1ba60d2a7dc2784d44: Status 404 returned error can't find the container with id a642e4639f062c0537f5e2a91b0edb2c1c4f1589c604ce1ba60d2a7dc2784d44 Jul 14 21:37:16.344016 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:16.280098 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd4582f_0e8a_4b90_a668_7f5260893841.slice/crio-93a1b04ae7bae82a343fa80b507c91dc932845f79639f18cfdc799fddea9539e WatchSource:0}: Error finding container 93a1b04ae7bae82a343fa80b507c91dc932845f79639f18cfdc799fddea9539e: Status 404 returned error can't find the container with id 93a1b04ae7bae82a343fa80b507c91dc932845f79639f18cfdc799fddea9539e Jul 14 21:37:16.344016 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.290497 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dctmt" Jul 14 21:37:16.344016 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:16.301758 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff48b38_946a_48b0_bf80_d46cee550a8b.slice/crio-897c8e9c0c7a9d1f5b5e72e5f123d2bfde9f9efeae34bc9dd1e8604a7cdb7090 WatchSource:0}: Error finding container 897c8e9c0c7a9d1f5b5e72e5f123d2bfde9f9efeae34bc9dd1e8604a7cdb7090: Status 404 returned error can't find the container with id 897c8e9c0c7a9d1f5b5e72e5f123d2bfde9f9efeae34bc9dd1e8604a7cdb7090 Jul 14 21:37:16.606089 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.606056 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lffmx\" (UniqueName: \"kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx\") pod \"network-check-target-8hxtk\" (UID: \"8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692\") " pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:16.606302 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.606117 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret\") pod \"global-pull-secret-syncer-f2b5s\" (UID: \"d0336426-165c-4987-beaa-f74fe715d480\") " pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:16.606302 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.606156 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs\") pod \"network-metrics-daemon-czmxj\" (UID: \"fab63e1e-b743-4ed6-bbbd-6686855f1a43\") " pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:16.606302 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:16.606258 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 21:37:16.606456 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:16.606333 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret podName:d0336426-165c-4987-beaa-f74fe715d480 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:17.60631841 +0000 UTC m=+4.145024060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret") pod "global-pull-secret-syncer-f2b5s" (UID: "d0336426-165c-4987-beaa-f74fe715d480") : object "kube-system"/"original-pull-secret" not registered Jul 14 21:37:16.606456 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:16.606258 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 21:37:16.606456 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:16.606388 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 21:37:16.606456 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:16.606404 2552 projected.go:196] Error preparing data for projected volume kube-api-access-lffmx for pod openshift-network-diagnostics/network-check-target-8hxtk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:37:16.606456 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:16.606266 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:37:16.606456 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:16.606457 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx podName:8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:17.606440064 +0000 UTC m=+4.145145730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lffmx" (UniqueName: "kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx") pod "network-check-target-8hxtk" (UID: "8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:37:16.606663 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:16.606498 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs podName:fab63e1e-b743-4ed6-bbbd-6686855f1a43 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:17.606471623 +0000 UTC m=+4.145177292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs") pod "network-metrics-daemon-czmxj" (UID: "fab63e1e-b743-4ed6-bbbd-6686855f1a43") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:37:16.991105 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.990990 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 21:32:15 +0000 UTC" deadline="2028-03-12 08:38:30.694774538 +0000 UTC" Jul 14 21:37:16.991105 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:16.991057 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14555h1m13.703721819s" Jul 14 21:37:17.015560 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:17.015423 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" event={"ID":"bbd4582f-0e8a-4b90-a668-7f5260893841","Type":"ContainerStarted","Data":"93a1b04ae7bae82a343fa80b507c91dc932845f79639f18cfdc799fddea9539e"} Jul 14 21:37:17.017933 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:17.017845 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9thwv" event={"ID":"451155f1-ad03-4be3-b198-895002675826","Type":"ContainerStarted","Data":"413418b0d30e090dc5a4dbf48829b364a8b75c89a2d26ac70c0267eac9bfbd62"} Jul 14 21:37:17.019742 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:17.019669 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c4vgt" event={"ID":"ba716d4b-af3d-44a4-a7d0-d134e8c805f3","Type":"ContainerStarted","Data":"5478904f8f649d1b629ceb6b492764c769901c46ab861bf6d56e1681a51c06c6"} Jul 14 21:37:17.021771 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:17.021693 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dctmt" event={"ID":"9ff48b38-946a-48b0-bf80-d46cee550a8b","Type":"ContainerStarted","Data":"897c8e9c0c7a9d1f5b5e72e5f123d2bfde9f9efeae34bc9dd1e8604a7cdb7090"} Jul 14 21:37:17.023763 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:17.023696 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p4twf" event={"ID":"6d5ffe2e-7323-41c6-a850-570f6a627f16","Type":"ContainerStarted","Data":"a642e4639f062c0537f5e2a91b0edb2c1c4f1589c604ce1ba60d2a7dc2784d44"} Jul 14 21:37:17.025235 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:17.025102 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2k4fl" event={"ID":"2a255c4c-c31b-4a1b-bd13-263359830473","Type":"ContainerStarted","Data":"1c76ae239571d97cc745d2857bb49d419df96cd92e2e82fe65f2b9e1a4711b39"} Jul 14 21:37:17.035844 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:17.035691 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" event={"ID":"239cdaed-5bca-4561-80be-9bac280ce4b3","Type":"ContainerStarted","Data":"de2dcad4e9ad5c733d58d679fb645ae3221b43cf21703658b7c10e6755a4b60f"} Jul 14 21:37:17.039323 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:17.039283 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2545q" event={"ID":"b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10","Type":"ContainerStarted","Data":"695e2af91620f2eb90664eafab6df02806e853c9c316c218ecc5fc2c1a7a126f"} Jul 14 21:37:17.040678 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:17.040651 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" event={"ID":"9cc0cd22-ce64-4aaf-8af0-2618da80d640","Type":"ContainerStarted","Data":"9bdeb0d12cac33598806bcf2008b83200c40c5e8448a3e2722a67ab417371c42"} Jul 14 21:37:17.614444 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:17.614404 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lffmx\" (UniqueName: \"kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx\") pod \"network-check-target-8hxtk\" (UID: \"8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692\") " pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:17.614634 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:17.614479 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret\") pod \"global-pull-secret-syncer-f2b5s\" (UID: \"d0336426-165c-4987-beaa-f74fe715d480\") " pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:17.614634 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:17.614509 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs\") pod \"network-metrics-daemon-czmxj\" (UID: \"fab63e1e-b743-4ed6-bbbd-6686855f1a43\") " pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:17.614743 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:17.614637 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:37:17.614743 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:17.614702 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs podName:fab63e1e-b743-4ed6-bbbd-6686855f1a43 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:19.614682996 +0000 UTC m=+6.153388649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs") pod "network-metrics-daemon-czmxj" (UID: "fab63e1e-b743-4ed6-bbbd-6686855f1a43") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:37:17.615163 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:17.615139 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 21:37:17.615163 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:17.615167 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 21:37:17.615337 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:17.615180 2552 projected.go:196] Error preparing data for projected volume kube-api-access-lffmx for pod openshift-network-diagnostics/network-check-target-8hxtk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:37:17.615337 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:17.615241 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx podName:8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:19.615226343 +0000 UTC m=+6.153931994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lffmx" (UniqueName: "kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx") pod "network-check-target-8hxtk" (UID: "8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:37:17.615337 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:17.615305 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 21:37:17.615337 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:17.615338 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret podName:d0336426-165c-4987-beaa-f74fe715d480 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:19.615327188 +0000 UTC m=+6.154032842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret") pod "global-pull-secret-syncer-f2b5s" (UID: "d0336426-165c-4987-beaa-f74fe715d480") : object "kube-system"/"original-pull-secret" not registered Jul 14 21:37:17.988328 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:17.988123 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:17.988328 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:17.988304 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2b5s" podUID="d0336426-165c-4987-beaa-f74fe715d480" Jul 14 21:37:17.988766 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:17.988746 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:17.988868 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:17.988847 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hxtk" podUID="8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692" Jul 14 21:37:17.988947 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:17.988925 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:17.989033 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:17.989016 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czmxj" podUID="fab63e1e-b743-4ed6-bbbd-6686855f1a43" Jul 14 21:37:17.991721 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:17.991672 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 21:32:15 +0000 UTC" deadline="2028-03-21 06:55:48.227107288 +0000 UTC" Jul 14 21:37:17.991721 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:17.991702 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14769h18m30.235409422s" Jul 14 21:37:19.630226 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:19.630161 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lffmx\" (UniqueName: \"kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx\") pod \"network-check-target-8hxtk\" (UID: \"8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692\") " pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:19.630707 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:19.630245 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret\") pod \"global-pull-secret-syncer-f2b5s\" (UID: \"d0336426-165c-4987-beaa-f74fe715d480\") " pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:19.630707 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:19.630284 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs\") pod \"network-metrics-daemon-czmxj\" (UID: \"fab63e1e-b743-4ed6-bbbd-6686855f1a43\") " pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:19.630707 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:19.630415 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:37:19.630707 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:19.630478 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs podName:fab63e1e-b743-4ed6-bbbd-6686855f1a43 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:23.630458106 +0000 UTC m=+10.169163759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs") pod "network-metrics-daemon-czmxj" (UID: "fab63e1e-b743-4ed6-bbbd-6686855f1a43") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:37:19.630916 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:19.630882 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 21:37:19.630916 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:19.630903 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 21:37:19.630916 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:19.630916 2552 projected.go:196] Error preparing data for projected volume kube-api-access-lffmx for pod openshift-network-diagnostics/network-check-target-8hxtk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:37:19.631117 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:19.630962 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx podName:8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:23.630946951 +0000 UTC m=+10.169652624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lffmx" (UniqueName: "kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx") pod "network-check-target-8hxtk" (UID: "8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:37:19.631117 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:19.631028 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 21:37:19.631117 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:19.631065 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret podName:d0336426-165c-4987-beaa-f74fe715d480 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:23.631051396 +0000 UTC m=+10.169757052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret") pod "global-pull-secret-syncer-f2b5s" (UID: "d0336426-165c-4987-beaa-f74fe715d480") : object "kube-system"/"original-pull-secret" not registered Jul 14 21:37:19.988485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:19.988070 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:19.988485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:19.988071 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:19.988485 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:19.988227 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2b5s" podUID="d0336426-165c-4987-beaa-f74fe715d480" Jul 14 21:37:19.988485 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:19.988308 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hxtk" podUID="8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692" Jul 14 21:37:19.988485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:19.988351 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:19.988485 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:19.988442 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czmxj" podUID="fab63e1e-b743-4ed6-bbbd-6686855f1a43" Jul 14 21:37:21.986335 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:21.986302 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:21.986692 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:21.986448 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czmxj" podUID="fab63e1e-b743-4ed6-bbbd-6686855f1a43" Jul 14 21:37:21.986888 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:21.986866 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:21.986988 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:21.986969 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2b5s" podUID="d0336426-165c-4987-beaa-f74fe715d480" Jul 14 21:37:21.987212 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:21.987133 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:21.987448 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:21.987389 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hxtk" podUID="8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692" Jul 14 21:37:23.664357 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:23.664319 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret\") pod \"global-pull-secret-syncer-f2b5s\" (UID: \"d0336426-165c-4987-beaa-f74fe715d480\") " pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:23.664751 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:23.664378 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs\") pod \"network-metrics-daemon-czmxj\" (UID: \"fab63e1e-b743-4ed6-bbbd-6686855f1a43\") " pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:23.664751 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:23.664440 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lffmx\" (UniqueName: \"kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx\") pod \"network-check-target-8hxtk\" (UID: \"8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692\") " pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:23.664751 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:23.664442 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 21:37:23.664751 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:23.664495 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:37:23.664751 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:23.664549 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs podName:fab63e1e-b743-4ed6-bbbd-6686855f1a43 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:31.664529437 +0000 UTC m=+18.203235102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs") pod "network-metrics-daemon-czmxj" (UID: "fab63e1e-b743-4ed6-bbbd-6686855f1a43") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:37:23.664751 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:23.664567 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret podName:d0336426-165c-4987-beaa-f74fe715d480 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:31.664556778 +0000 UTC m=+18.203262433 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret") pod "global-pull-secret-syncer-f2b5s" (UID: "d0336426-165c-4987-beaa-f74fe715d480") : object "kube-system"/"original-pull-secret" not registered Jul 14 21:37:23.664751 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:23.664642 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 21:37:23.664751 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:23.664657 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 21:37:23.664751 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:23.664669 2552 projected.go:196] Error preparing data for projected volume kube-api-access-lffmx for pod openshift-network-diagnostics/network-check-target-8hxtk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:37:23.664751 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:23.664714 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx podName:8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:31.664703297 +0000 UTC m=+18.203408969 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lffmx" (UniqueName: "kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx") pod "network-check-target-8hxtk" (UID: "8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:37:23.987600 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:23.987089 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:23.987600 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:23.987218 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czmxj" podUID="fab63e1e-b743-4ed6-bbbd-6686855f1a43" Jul 14 21:37:23.987600 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:23.987261 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:23.987600 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:23.987364 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2b5s" podUID="d0336426-165c-4987-beaa-f74fe715d480" Jul 14 21:37:23.987600 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:23.987411 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:23.987600 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:23.987529 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hxtk" podUID="8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692" Jul 14 21:37:25.988895 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:25.988060 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:25.988895 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:25.988224 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czmxj" podUID="fab63e1e-b743-4ed6-bbbd-6686855f1a43" Jul 14 21:37:25.988895 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:25.988631 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:25.988895 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:25.988731 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2b5s" podUID="d0336426-165c-4987-beaa-f74fe715d480" Jul 14 21:37:25.988895 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:25.988787 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:25.988895 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:25.988851 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hxtk" podUID="8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692" Jul 14 21:37:27.986352 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:27.986294 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:27.986803 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:27.986416 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:27.986803 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:27.986427 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hxtk" podUID="8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692" Jul 14 21:37:27.986803 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:27.986447 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:27.986803 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:27.986545 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czmxj" podUID="fab63e1e-b743-4ed6-bbbd-6686855f1a43" Jul 14 21:37:27.986803 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:27.986613 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2b5s" podUID="d0336426-165c-4987-beaa-f74fe715d480" Jul 14 21:37:29.986172 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:29.986136 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:29.986634 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:29.986136 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:29.986634 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:29.986136 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:29.986634 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:29.986282 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hxtk" podUID="8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692" Jul 14 21:37:29.986634 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:29.986358 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czmxj" podUID="fab63e1e-b743-4ed6-bbbd-6686855f1a43" Jul 14 21:37:29.986634 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:29.986447 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2b5s" podUID="d0336426-165c-4987-beaa-f74fe715d480" Jul 14 21:37:31.721963 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:31.721922 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret\") pod \"global-pull-secret-syncer-f2b5s\" (UID: \"d0336426-165c-4987-beaa-f74fe715d480\") " pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:31.722429 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:31.721977 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs\") pod \"network-metrics-daemon-czmxj\" (UID: \"fab63e1e-b743-4ed6-bbbd-6686855f1a43\") " pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:31.722429 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:31.722022 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lffmx\" (UniqueName: \"kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx\") pod \"network-check-target-8hxtk\" (UID: \"8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692\") " pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:31.722429 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:31.722103 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 21:37:31.722429 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:31.722125 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:37:31.722429 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:31.722147 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 21:37:31.722429 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:31.722161 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 21:37:31.722429 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:31.722170 2552 projected.go:196] Error preparing data for projected volume kube-api-access-lffmx for pod openshift-network-diagnostics/network-check-target-8hxtk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:37:31.722429 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:31.722176 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret podName:d0336426-165c-4987-beaa-f74fe715d480 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:47.722160736 +0000 UTC m=+34.260866402 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret") pod "global-pull-secret-syncer-f2b5s" (UID: "d0336426-165c-4987-beaa-f74fe715d480") : object "kube-system"/"original-pull-secret" not registered Jul 14 21:37:31.722429 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:31.722207 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs podName:fab63e1e-b743-4ed6-bbbd-6686855f1a43 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:47.722185644 +0000 UTC m=+34.260891294 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs") pod "network-metrics-daemon-czmxj" (UID: "fab63e1e-b743-4ed6-bbbd-6686855f1a43") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:37:31.722429 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:31.722224 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx podName:8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:47.722213267 +0000 UTC m=+34.260918916 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lffmx" (UniqueName: "kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx") pod "network-check-target-8hxtk" (UID: "8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:37:31.986441 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:31.986274 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:31.986441 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:31.986404 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2b5s" podUID="d0336426-165c-4987-beaa-f74fe715d480" Jul 14 21:37:31.986441 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:31.986426 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:31.986721 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:31.986274 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:31.986721 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:31.986595 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czmxj" podUID="fab63e1e-b743-4ed6-bbbd-6686855f1a43" Jul 14 21:37:31.986721 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:31.986681 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hxtk" podUID="8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692" Jul 14 21:37:33.989413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:33.988380 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:33.989413 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:33.988537 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2b5s" podUID="d0336426-165c-4987-beaa-f74fe715d480" Jul 14 21:37:33.989413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:33.989083 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:33.989413 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:33.989216 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hxtk" podUID="8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692" Jul 14 21:37:33.989413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:33.989280 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:33.989413 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:33.989390 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czmxj" podUID="fab63e1e-b743-4ed6-bbbd-6686855f1a43" Jul 14 21:37:35.986260 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:35.985948 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:35.986783 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:35.985948 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:35.986783 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:35.986339 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2b5s" podUID="d0336426-165c-4987-beaa-f74fe715d480" Jul 14 21:37:35.986783 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:35.986429 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czmxj" podUID="fab63e1e-b743-4ed6-bbbd-6686855f1a43" Jul 14 21:37:35.986783 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:35.985998 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:35.986783 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:35.986506 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hxtk" podUID="8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692" Jul 14 21:37:36.085234 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.085183 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p4twf" event={"ID":"6d5ffe2e-7323-41c6-a850-570f6a627f16","Type":"ContainerStarted","Data":"63caef2990d7e9d80a588edf2b552554214ec502ebb22cc87c66c5ca4360032f"} Jul 14 21:37:36.086619 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.086581 2552 generic.go:356] "Generic (PLEG): container finished" podID="2a255c4c-c31b-4a1b-bd13-263359830473" containerID="861908b3189b826e58160d6eb94a7ceb6dfb90690c400132f4a7da2eee58850b" exitCode=0 Jul 14 21:37:36.086732 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.086662 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2k4fl" event={"ID":"2a255c4c-c31b-4a1b-bd13-263359830473","Type":"ContainerDied","Data":"861908b3189b826e58160d6eb94a7ceb6dfb90690c400132f4a7da2eee58850b"} Jul 14 21:37:36.088155 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.088136 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" event={"ID":"239cdaed-5bca-4561-80be-9bac280ce4b3","Type":"ContainerStarted","Data":"58929d2814855acd602e61b3d721b4d69de7b8904362ebc0387753fdf346b819"} Jul 14 21:37:36.089844 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.089790 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2545q" event={"ID":"b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10","Type":"ContainerStarted","Data":"68cb26b4859e3e39067ec0115ca050d0930bba8f726898cd7feaa94606e359d5"} Jul 14 21:37:36.091981 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.091964 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 21:37:36.092332 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.092311 2552 generic.go:356] "Generic (PLEG): container finished" podID="9cc0cd22-ce64-4aaf-8af0-2618da80d640" containerID="b5b08b87f42a95b0c2cc091616f03f169fe3505b14615fa50eb16d925631f5bd" exitCode=1 Jul 14 21:37:36.092419 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.092374 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" event={"ID":"9cc0cd22-ce64-4aaf-8af0-2618da80d640","Type":"ContainerStarted","Data":"941f50895d4b330be61d53e5dcf6a6b0d5c0d22c2aa0b9c2c9966cac203871f5"} Jul 14 21:37:36.092458 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.092435 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" event={"ID":"9cc0cd22-ce64-4aaf-8af0-2618da80d640","Type":"ContainerStarted","Data":"725eb82dcae9c16e5767bf8473865c115a887e291f8f6b55d9885402654645d3"} Jul 14 21:37:36.092487 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.092473 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" event={"ID":"9cc0cd22-ce64-4aaf-8af0-2618da80d640","Type":"ContainerDied","Data":"b5b08b87f42a95b0c2cc091616f03f169fe3505b14615fa50eb16d925631f5bd"} Jul 14 21:37:36.092538 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.092512 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" event={"ID":"9cc0cd22-ce64-4aaf-8af0-2618da80d640","Type":"ContainerStarted","Data":"990b8272c4ef8eef3f1b41902192298436d88c146ef719fe7df39296c976a371"} Jul 14 21:37:36.093973 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.093954 2552 generic.go:356] "Generic (PLEG): container finished" podID="2a59ded2d15e5a27ff495bdc28701515" containerID="261bfe17064bc0bfcf91f9df4c25185db9d4607c205b15e3bffa0c53859b79df" exitCode=0 Jul 14 21:37:36.094060 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.094031 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal" event={"ID":"2a59ded2d15e5a27ff495bdc28701515","Type":"ContainerDied","Data":"261bfe17064bc0bfcf91f9df4c25185db9d4607c205b15e3bffa0c53859b79df"} Jul 14 21:37:36.095571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.095552 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal" event={"ID":"6347ca5f52a32e5fefe18e2f2e47b4f4","Type":"ContainerStarted","Data":"e9b766bbd216062f8f03ef7f6cb25e03907ac34c8993da25f1799a7d5d247ac3"} Jul 14 21:37:36.098953 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.098931 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" event={"ID":"bbd4582f-0e8a-4b90-a668-7f5260893841","Type":"ContainerStarted","Data":"35c0f292df96ce53d75c4d8084f846ac27881e597774174bd545e8c133d4af89"} Jul 14 21:37:36.100067 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.100047 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9thwv" event={"ID":"451155f1-ad03-4be3-b198-895002675826","Type":"ContainerStarted","Data":"669b2ac1c69c589ee069b656fec3d70ccef855fb57aec0fddd467760b7278fc4"} Jul 14 21:37:36.101245 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.101223 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c4vgt" event={"ID":"ba716d4b-af3d-44a4-a7d0-d134e8c805f3","Type":"ContainerStarted","Data":"dafa1a2cdc0ff62696044b81603a14d30793b23a27189410fcbe7b951ee8df88"} Jul 14 21:37:36.106439 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.106392 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p4twf" podStartSLOduration=2.812476523 podStartE2EDuration="22.106380516s" totalImagesPullingTime="19.293903993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:37:36.106057981 +0000 UTC m=+22.644763654" watchObservedRunningTime="2026-07-14 21:37:36.106380516 +0000 UTC m=+22.645086187" Jul 14 21:37:36.121660 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.121620 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal" podStartSLOduration=21.121607848 podStartE2EDuration="21.121607848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:37:36.121233595 +0000 UTC m=+22.659939263" watchObservedRunningTime="2026-07-14 21:37:36.121607848 +0000 UTC m=+22.660313520" Jul 14 21:37:36.140047 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.140001 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xnl8n" podStartSLOduration=2.959421526 podStartE2EDuration="22.139981719s" totalImagesPullingTime="19.180560193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:37:36.139909269 +0000 UTC m=+22.678614945" watchObservedRunningTime="2026-07-14 21:37:36.139981719 +0000 UTC m=+22.678687393" Jul 14 21:37:36.204248 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.204183 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9thwv" podStartSLOduration=3.050566074 podStartE2EDuration="22.204168375s" totalImagesPullingTime="19.153602301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:37:36.204092699 +0000 UTC m=+22.742798372" watchObservedRunningTime="2026-07-14 21:37:36.204168375 +0000 UTC m=+22.742874046" Jul 14 21:37:36.285851 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.285797 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-c4vgt" podStartSLOduration=3.120783042 podStartE2EDuration="22.285781817s" totalImagesPullingTime="19.164998775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:37:36.261481471 +0000 UTC m=+22.800187165" watchObservedRunningTime="2026-07-14 21:37:36.285781817 +0000 UTC m=+22.824487489" Jul 14 21:37:36.286031 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:36.285895 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2545q" podStartSLOduration=7.834362143 podStartE2EDuration="22.285886654s" totalImagesPullingTime="14.451524511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:37:36.285501694 +0000 UTC m=+22.824207366" watchObservedRunningTime="2026-07-14 21:37:36.285886654 +0000 UTC m=+22.824592327" Jul 14 21:37:37.107821 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:37.107795 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 21:37:37.108357 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:37.108182 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" event={"ID":"9cc0cd22-ce64-4aaf-8af0-2618da80d640","Type":"ContainerStarted","Data":"417a06d26a697c077d8616c3c196d8692ee3609195e8ae99969dd33b4a6d122b"} Jul 14 21:37:37.108357 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:37.108333 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" event={"ID":"9cc0cd22-ce64-4aaf-8af0-2618da80d640","Type":"ContainerStarted","Data":"0dc6f4a5fa9d6ac9d75c3b3ba15a7b037b15f43e0d9ad9205a4bafda29e6b095"} Jul 14 21:37:37.110967 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:37.110939 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal" event={"ID":"2a59ded2d15e5a27ff495bdc28701515","Type":"ContainerStarted","Data":"28c1e4b8005f4b4b4a49e4dcee2841f9e300adbe08223f2bca35ff4edf09035d"} Jul 14 21:37:37.112430 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:37.112394 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dctmt" event={"ID":"9ff48b38-946a-48b0-bf80-d46cee550a8b","Type":"ContainerStarted","Data":"df55b8bc12aba15f1357d5d8c27ad658c563c177eeabc06d0ef94d4f7601f56b"} Jul 14 21:37:37.166085 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:37.166040 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dctmt" podStartSLOduration=4.036012415 podStartE2EDuration="23.166024837s" totalImagesPullingTime="19.130012422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:37:37.165997508 +0000 UTC m=+23.704703182" watchObservedRunningTime="2026-07-14 21:37:37.166024837 +0000 UTC m=+23.704730508" Jul 14 21:37:37.166525 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:37.166495 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal" podStartSLOduration=22.166485883 podStartE2EDuration="22.166485883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:37:37.139576295 +0000 UTC m=+23.678281972" watchObservedRunningTime="2026-07-14 21:37:37.166485883 +0000 UTC m=+23.705191553" Jul 14 21:37:37.207597 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:37.207568 2552 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 21:37:37.892075 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:37.892034 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9thwv" Jul 14 21:37:37.892644 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:37.892624 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9thwv" Jul 14 21:37:37.986271 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:37.986237 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:37.986465 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:37.986245 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:37.986465 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:37.986373 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czmxj" podUID="fab63e1e-b743-4ed6-bbbd-6686855f1a43" Jul 14 21:37:37.986465 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:37.986247 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:37.986465 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:37.986442 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hxtk" podUID="8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692" Jul 14 21:37:37.986642 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:37.986502 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2b5s" podUID="d0336426-165c-4987-beaa-f74fe715d480" Jul 14 21:37:38.002963 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:38.002852 2552 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T21:37:37.207591887Z","UUID":"55413f68-f281-400c-8c07-54aa9969909d","Handler":null,"Name":"","Endpoint":""} Jul 14 21:37:38.004450 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:38.004427 2552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 21:37:38.004605 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:38.004473 2552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 21:37:38.116815 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:38.116563 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" event={"ID":"bbd4582f-0e8a-4b90-a668-7f5260893841","Type":"ContainerStarted","Data":"2b2cdf9755cd2be79e7b9b2f316418f00b31e1041210765ba527996b320d8449"} Jul 14 21:37:38.117292 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:38.116843 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9thwv" Jul 14 21:37:38.117292 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:38.117240 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9thwv" Jul 14 21:37:39.122395 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:39.122374 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 21:37:39.122886 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:39.122837 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" event={"ID":"9cc0cd22-ce64-4aaf-8af0-2618da80d640","Type":"ContainerStarted","Data":"a6edb7940ea45af4bb3efbf8d3c8fa65d432f1bca63b0b686a12c5b6875cd435"} Jul 14 21:37:39.125091 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:39.124953 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" event={"ID":"bbd4582f-0e8a-4b90-a668-7f5260893841","Type":"ContainerStarted","Data":"6995678b8a125e6cea7fb6934d9fcc001b9307f2006c7b06188e2f22b0005398"} Jul 14 21:37:39.147025 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:39.146967 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6srfg" podStartSLOduration=3.018717453 podStartE2EDuration="25.146945136s" totalImagesPullingTime="22.128227683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:37:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:37:39.145747637 +0000 UTC m=+25.684453306" watchObservedRunningTime="2026-07-14 21:37:39.146945136 +0000 UTC m=+25.685650810" Jul 14 21:37:39.985837 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:39.985797 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:39.985837 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:39.985832 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:39.986089 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:39.985797 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:39.986089 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:39.985954 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czmxj" podUID="fab63e1e-b743-4ed6-bbbd-6686855f1a43" Jul 14 21:37:39.986089 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:39.986034 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2b5s" podUID="d0336426-165c-4987-beaa-f74fe715d480" Jul 14 21:37:39.986267 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:39.986122 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hxtk" podUID="8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692" Jul 14 21:37:41.133160 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:41.133137 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 21:37:41.133797 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:41.133551 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" event={"ID":"9cc0cd22-ce64-4aaf-8af0-2618da80d640","Type":"ContainerStarted","Data":"4beb942e7f69bda2c7847bd8af6c8cef33553993baa7f60e9acddef6739c3748"} Jul 14 21:37:41.133840 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:41.133807 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:41.133893 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:41.133879 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:41.133998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:41.133983 2552 scope.go:117] "RemoveContainer" containerID="b5b08b87f42a95b0c2cc091616f03f169fe3505b14615fa50eb16d925631f5bd" Jul 14 21:37:41.149935 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:41.149912 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:41.985899 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:41.985859 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:41.986105 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:41.985925 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:41.986105 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:41.985859 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:41.986105 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:41.986010 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czmxj" podUID="fab63e1e-b743-4ed6-bbbd-6686855f1a43" Jul 14 21:37:41.986105 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:41.986073 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2b5s" podUID="d0336426-165c-4987-beaa-f74fe715d480" Jul 14 21:37:41.986357 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:41.986139 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hxtk" podUID="8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692" Jul 14 21:37:42.140986 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:42.140958 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 21:37:42.141620 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:42.141475 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" event={"ID":"9cc0cd22-ce64-4aaf-8af0-2618da80d640","Type":"ContainerStarted","Data":"3c73cf41e424c1e4b11dc48a00c4c092a839304a9b2fdea86cbac26e2ebd71ef"} Jul 14 21:37:42.141793 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:42.141768 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:42.143971 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:42.143940 2552 generic.go:356] "Generic (PLEG): container finished" podID="2a255c4c-c31b-4a1b-bd13-263359830473" containerID="b898d6be4b1d7685aa0d72290497a908a134a3697f3c9eb3f7d45ec10beabe87" exitCode=0 Jul 14 21:37:42.144092 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:42.143986 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2k4fl" event={"ID":"2a255c4c-c31b-4a1b-bd13-263359830473","Type":"ContainerDied","Data":"b898d6be4b1d7685aa0d72290497a908a134a3697f3c9eb3f7d45ec10beabe87"} Jul 14 21:37:42.163334 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:42.163156 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:37:42.177811 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:42.177753 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" podStartSLOduration=8.928986519 podStartE2EDuration="28.177732061s" totalImagesPullingTime="19.248745542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:37:42.177636672 +0000 UTC m=+28.716342345" watchObservedRunningTime="2026-07-14 21:37:42.177732061 +0000 UTC m=+28.716437734" Jul 14 21:37:43.337529 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:43.337302 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f2b5s"] Jul 14 21:37:43.337915 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:43.337636 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:43.337915 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:43.337735 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2b5s" podUID="d0336426-165c-4987-beaa-f74fe715d480" Jul 14 21:37:43.340877 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:43.340845 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-czmxj"] Jul 14 21:37:43.340994 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:43.340965 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:43.341103 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:43.341082 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czmxj" podUID="fab63e1e-b743-4ed6-bbbd-6686855f1a43" Jul 14 21:37:43.353324 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:43.353296 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8hxtk"] Jul 14 21:37:43.353455 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:43.353389 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:43.353513 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:43.353452 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hxtk" podUID="8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692" Jul 14 21:37:44.150946 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:44.150911 2552 generic.go:356] "Generic (PLEG): container finished" podID="2a255c4c-c31b-4a1b-bd13-263359830473" containerID="2d02a3b24d19280e0dcb33dd0f22e284cd69a9cbd48a84408895364cf470d71f" exitCode=0 Jul 14 21:37:44.151156 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:44.150991 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2k4fl" event={"ID":"2a255c4c-c31b-4a1b-bd13-263359830473","Type":"ContainerDied","Data":"2d02a3b24d19280e0dcb33dd0f22e284cd69a9cbd48a84408895364cf470d71f"} Jul 14 21:37:44.986237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:44.986178 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:44.986237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:44.986236 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:44.986751 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:44.986178 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:44.986751 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:44.986337 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czmxj" podUID="fab63e1e-b743-4ed6-bbbd-6686855f1a43" Jul 14 21:37:44.986751 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:44.986403 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2b5s" podUID="d0336426-165c-4987-beaa-f74fe715d480" Jul 14 21:37:44.986751 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:44.986466 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hxtk" podUID="8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692" Jul 14 21:37:45.156003 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:45.155964 2552 generic.go:356] "Generic (PLEG): container finished" podID="2a255c4c-c31b-4a1b-bd13-263359830473" containerID="c7b1dc64a936664803127e5faa79b3f524223f07890d8c13ae985c1ec28b0969" exitCode=0 Jul 14 21:37:45.156141 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:45.156041 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2k4fl" event={"ID":"2a255c4c-c31b-4a1b-bd13-263359830473","Type":"ContainerDied","Data":"c7b1dc64a936664803127e5faa79b3f524223f07890d8c13ae985c1ec28b0969"} Jul 14 21:37:46.985949 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:46.985911 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:46.986685 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:46.985911 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:46.986685 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:46.986038 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-czmxj" podUID="fab63e1e-b743-4ed6-bbbd-6686855f1a43" Jul 14 21:37:46.986685 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:46.986126 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2b5s" podUID="d0336426-165c-4987-beaa-f74fe715d480" Jul 14 21:37:46.986685 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:46.985911 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:46.986685 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:46.986226 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hxtk" podUID="8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692" Jul 14 21:37:47.739614 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:47.739575 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lffmx\" (UniqueName: \"kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx\") pod \"network-check-target-8hxtk\" (UID: \"8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692\") " pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:47.739807 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:47.739652 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret\") pod \"global-pull-secret-syncer-f2b5s\" (UID: \"d0336426-165c-4987-beaa-f74fe715d480\") " pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:47.739807 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:47.739683 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs\") pod \"network-metrics-daemon-czmxj\" (UID: \"fab63e1e-b743-4ed6-bbbd-6686855f1a43\") " pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:47.739807 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:47.739758 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 21:37:47.739807 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:47.739770 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:37:47.739807 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:47.739779 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 21:37:47.739807 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:47.739791 2552 projected.go:196] Error preparing data for projected volume kube-api-access-lffmx for pod openshift-network-diagnostics/network-check-target-8hxtk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:37:47.739807 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:47.739790 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 21:37:47.740096 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:47.739833 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx podName:8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:19.739816849 +0000 UTC m=+66.278522511 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lffmx" (UniqueName: "kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx") pod "network-check-target-8hxtk" (UID: "8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 21:37:47.740096 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:47.739851 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs podName:fab63e1e-b743-4ed6-bbbd-6686855f1a43 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:19.739842824 +0000 UTC m=+66.278548474 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs") pod "network-metrics-daemon-czmxj" (UID: "fab63e1e-b743-4ed6-bbbd-6686855f1a43") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 21:37:47.740096 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:47.739864 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret podName:d0336426-165c-4987-beaa-f74fe715d480 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:19.73985696 +0000 UTC m=+66.278562610 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret") pod "global-pull-secret-syncer-f2b5s" (UID: "d0336426-165c-4987-beaa-f74fe715d480") : object "kube-system"/"original-pull-secret" not registered Jul 14 21:37:48.274936 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.274855 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeReady" Jul 14 21:37:48.275348 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.275001 2552 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 21:37:48.319875 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.319841 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cf444f6dc-f2bc5"] Jul 14 21:37:48.322520 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.322497 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.325991 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.325816 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 14 21:37:48.326133 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.325887 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 14 21:37:48.326452 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.326434 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 14 21:37:48.326670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.326652 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tcngh\"" Jul 14 21:37:48.332390 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.332370 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 14 21:37:48.346124 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.346097 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cf444f6dc-f2bc5"] Jul 14 21:37:48.363759 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.363723 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xxhmt"] Jul 14 21:37:48.365657 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.365626 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jrvlv"] Jul 14 21:37:48.365784 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.365762 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xxhmt" Jul 14 21:37:48.369419 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.368906 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jrvlv" Jul 14 21:37:48.370572 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.370545 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cq9kr\"" Jul 14 21:37:48.370686 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.370544 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 21:37:48.370686 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.370605 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 21:37:48.370946 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.370927 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 21:37:48.374561 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.374374 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 21:37:48.374561 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.374384 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zdsh8\"" Jul 14 21:37:48.375989 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.375972 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 21:37:48.381277 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.381258 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xxhmt"] Jul 14 21:37:48.403679 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.403653 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jrvlv"] Jul 14 21:37:48.445081 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.445044 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjhtb\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-kube-api-access-bjhtb\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.445081 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.445078 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sq4m\" (UniqueName: \"kubernetes.io/projected/9687cbd1-86bc-4429-9f4d-a8cda7399acb-kube-api-access-2sq4m\") pod \"ingress-canary-xxhmt\" (UID: \"9687cbd1-86bc-4429-9f4d-a8cda7399acb\") " pod="openshift-ingress-canary/ingress-canary-xxhmt" Jul 14 21:37:48.445318 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.445124 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/547803e8-dd34-48fb-bb45-f08182623df4-image-registry-private-configuration\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.445318 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.445152 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/547803e8-dd34-48fb-bb45-f08182623df4-registry-certificates\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.445318 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.445216 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/547803e8-dd34-48fb-bb45-f08182623df4-trusted-ca\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.445318 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.445253 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.445318 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.445275 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e461f5b-ff8f-4018-a3d2-9de160cabffd-config-volume\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:37:48.445318 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.445297 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gvqp\" (UniqueName: \"kubernetes.io/projected/4e461f5b-ff8f-4018-a3d2-9de160cabffd-kube-api-access-5gvqp\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:37:48.445548 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.445321 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/547803e8-dd34-48fb-bb45-f08182623df4-installation-pull-secrets\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.445548 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.445376 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-bound-sa-token\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.445548 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.445397 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9687cbd1-86bc-4429-9f4d-a8cda7399acb-cert\") pod \"ingress-canary-xxhmt\" (UID: \"9687cbd1-86bc-4429-9f4d-a8cda7399acb\") " pod="openshift-ingress-canary/ingress-canary-xxhmt" Jul 14 21:37:48.445548 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.445447 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:37:48.445548 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.445462 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/547803e8-dd34-48fb-bb45-f08182623df4-ca-trust-extracted\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.445548 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.445489 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e461f5b-ff8f-4018-a3d2-9de160cabffd-tmp-dir\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:37:48.546330 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.546173 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:37:48.546330 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.546232 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/547803e8-dd34-48fb-bb45-f08182623df4-ca-trust-extracted\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.546330 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.546275 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e461f5b-ff8f-4018-a3d2-9de160cabffd-tmp-dir\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:37:48.546330 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.546319 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjhtb\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-kube-api-access-bjhtb\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.546645 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:48.546352 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 21:37:48.546645 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:48.546427 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls podName:4e461f5b-ff8f-4018-a3d2-9de160cabffd nodeName:}" failed. No retries permitted until 2026-07-14 21:37:49.046405288 +0000 UTC m=+35.585110938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls") pod "dns-default-jrvlv" (UID: "4e461f5b-ff8f-4018-a3d2-9de160cabffd") : secret "dns-default-metrics-tls" not found Jul 14 21:37:48.546645 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.546354 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sq4m\" (UniqueName: \"kubernetes.io/projected/9687cbd1-86bc-4429-9f4d-a8cda7399acb-kube-api-access-2sq4m\") pod \"ingress-canary-xxhmt\" (UID: \"9687cbd1-86bc-4429-9f4d-a8cda7399acb\") " pod="openshift-ingress-canary/ingress-canary-xxhmt" Jul 14 21:37:48.546645 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.546543 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/547803e8-dd34-48fb-bb45-f08182623df4-image-registry-private-configuration\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.546645 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.546580 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/547803e8-dd34-48fb-bb45-f08182623df4-registry-certificates\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.546645 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.546611 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/547803e8-dd34-48fb-bb45-f08182623df4-trusted-ca\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.546937 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.546722 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/547803e8-dd34-48fb-bb45-f08182623df4-ca-trust-extracted\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.546937 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.546748 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.546937 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.546783 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e461f5b-ff8f-4018-a3d2-9de160cabffd-config-volume\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:37:48.546937 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.546723 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e461f5b-ff8f-4018-a3d2-9de160cabffd-tmp-dir\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:37:48.546937 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:48.546828 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 21:37:48.546937 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:48.546844 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cf444f6dc-f2bc5: secret "image-registry-tls" not found Jul 14 21:37:48.546937 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:48.546921 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls podName:547803e8-dd34-48fb-bb45-f08182623df4 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:49.046902579 +0000 UTC m=+35.585608248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls") pod "image-registry-cf444f6dc-f2bc5" (UID: "547803e8-dd34-48fb-bb45-f08182623df4") : secret "image-registry-tls" not found Jul 14 21:37:48.546937 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.546806 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gvqp\" (UniqueName: \"kubernetes.io/projected/4e461f5b-ff8f-4018-a3d2-9de160cabffd-kube-api-access-5gvqp\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:37:48.547326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.546960 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/547803e8-dd34-48fb-bb45-f08182623df4-installation-pull-secrets\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.547326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.546992 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-bound-sa-token\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.547326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.547165 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/547803e8-dd34-48fb-bb45-f08182623df4-registry-certificates\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.547326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.547240 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9687cbd1-86bc-4429-9f4d-a8cda7399acb-cert\") pod \"ingress-canary-xxhmt\" (UID: \"9687cbd1-86bc-4429-9f4d-a8cda7399acb\") " pod="openshift-ingress-canary/ingress-canary-xxhmt" Jul 14 21:37:48.547547 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.547344 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e461f5b-ff8f-4018-a3d2-9de160cabffd-config-volume\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:37:48.547649 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.547628 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/547803e8-dd34-48fb-bb45-f08182623df4-trusted-ca\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.551562 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.551536 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/547803e8-dd34-48fb-bb45-f08182623df4-installation-pull-secrets\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.551702 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.551568 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/547803e8-dd34-48fb-bb45-f08182623df4-image-registry-private-configuration\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.551702 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.551576 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9687cbd1-86bc-4429-9f4d-a8cda7399acb-cert\") pod \"ingress-canary-xxhmt\" (UID: \"9687cbd1-86bc-4429-9f4d-a8cda7399acb\") " pod="openshift-ingress-canary/ingress-canary-xxhmt" Jul 14 21:37:48.559349 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.559323 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sq4m\" (UniqueName: \"kubernetes.io/projected/9687cbd1-86bc-4429-9f4d-a8cda7399acb-kube-api-access-2sq4m\") pod \"ingress-canary-xxhmt\" (UID: \"9687cbd1-86bc-4429-9f4d-a8cda7399acb\") " pod="openshift-ingress-canary/ingress-canary-xxhmt" Jul 14 21:37:48.565424 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.565388 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gvqp\" (UniqueName: \"kubernetes.io/projected/4e461f5b-ff8f-4018-a3d2-9de160cabffd-kube-api-access-5gvqp\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:37:48.566646 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.566620 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-bound-sa-token\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.567364 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.567338 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjhtb\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-kube-api-access-bjhtb\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:48.600617 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.600576 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:37:48.680497 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.680463 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xxhmt" Jul 14 21:37:48.722824 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.722787 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:37:48.986051 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.986018 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:37:48.986051 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.986059 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:37:48.986325 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.986059 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:37:48.990310 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.990284 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 21:37:48.990679 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.990654 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bkps7\"" Jul 14 21:37:48.992093 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.992071 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 21:37:48.992253 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.992182 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2rjnp\"" Jul 14 21:37:48.992585 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.992553 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 21:37:48.993061 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:48.993042 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 21:37:49.050376 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:49.050344 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:49.050552 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:49.050404 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:37:49.050552 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:49.050530 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 21:37:49.050552 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:49.050548 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cf444f6dc-f2bc5: secret "image-registry-tls" not found Jul 14 21:37:49.050680 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:49.050596 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 21:37:49.050680 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:49.050602 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls podName:547803e8-dd34-48fb-bb45-f08182623df4 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:50.050587166 +0000 UTC m=+36.589292841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls") pod "image-registry-cf444f6dc-f2bc5" (UID: "547803e8-dd34-48fb-bb45-f08182623df4") : secret "image-registry-tls" not found Jul 14 21:37:49.050680 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:49.050679 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls podName:4e461f5b-ff8f-4018-a3d2-9de160cabffd nodeName:}" failed. No retries permitted until 2026-07-14 21:37:50.050660973 +0000 UTC m=+36.589366632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls") pod "dns-default-jrvlv" (UID: "4e461f5b-ff8f-4018-a3d2-9de160cabffd") : secret "dns-default-metrics-tls" not found Jul 14 21:37:49.609620 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:49.609558 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xxhmt"] Jul 14 21:37:49.657902 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:49.657865 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9687cbd1_86bc_4429_9f4d_a8cda7399acb.slice/crio-dd27591cc204e75a11b227972322a27e6a6d17f7c8aea8fe2f41aa40bb7e4f8c WatchSource:0}: Error finding container dd27591cc204e75a11b227972322a27e6a6d17f7c8aea8fe2f41aa40bb7e4f8c: Status 404 returned error can't find the container with id dd27591cc204e75a11b227972322a27e6a6d17f7c8aea8fe2f41aa40bb7e4f8c Jul 14 21:37:49.813100 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:49.813062 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:37:49.914482 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:49.914448 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:37:50.059580 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:50.059502 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:50.059580 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:50.059546 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:37:50.059765 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:50.059662 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 21:37:50.059765 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:50.059664 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 21:37:50.059765 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:50.059684 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cf444f6dc-f2bc5: secret "image-registry-tls" not found Jul 14 21:37:50.059765 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:50.059716 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls podName:4e461f5b-ff8f-4018-a3d2-9de160cabffd nodeName:}" failed. No retries permitted until 2026-07-14 21:37:52.059700463 +0000 UTC m=+38.598406112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls") pod "dns-default-jrvlv" (UID: "4e461f5b-ff8f-4018-a3d2-9de160cabffd") : secret "dns-default-metrics-tls" not found Jul 14 21:37:50.059765 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:50.059734 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls podName:547803e8-dd34-48fb-bb45-f08182623df4 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:52.05972134 +0000 UTC m=+38.598427008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls") pod "image-registry-cf444f6dc-f2bc5" (UID: "547803e8-dd34-48fb-bb45-f08182623df4") : secret "image-registry-tls" not found Jul 14 21:37:50.169981 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:50.169943 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xxhmt" event={"ID":"9687cbd1-86bc-4429-9f4d-a8cda7399acb","Type":"ContainerStarted","Data":"dd27591cc204e75a11b227972322a27e6a6d17f7c8aea8fe2f41aa40bb7e4f8c"} Jul 14 21:37:50.173048 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:50.173020 2552 generic.go:356] "Generic (PLEG): container finished" podID="2a255c4c-c31b-4a1b-bd13-263359830473" containerID="3fc0574032c3ac4254665e21540528807391adfc70b5a26f4a3800b6773fb433" exitCode=0 Jul 14 21:37:50.173224 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:50.173073 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2k4fl" event={"ID":"2a255c4c-c31b-4a1b-bd13-263359830473","Type":"ContainerDied","Data":"3fc0574032c3ac4254665e21540528807391adfc70b5a26f4a3800b6773fb433"} Jul 14 21:37:51.036921 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:51.036734 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:37:51.179272 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:51.179241 2552 generic.go:356] "Generic (PLEG): container finished" podID="2a255c4c-c31b-4a1b-bd13-263359830473" containerID="4b8e09008d10ea090ec09671053bdaafcce065235a9308e1cf7b330f87c1f1ec" exitCode=0 Jul 14 21:37:51.179436 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:51.179295 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2k4fl" event={"ID":"2a255c4c-c31b-4a1b-bd13-263359830473","Type":"ContainerDied","Data":"4b8e09008d10ea090ec09671053bdaafcce065235a9308e1cf7b330f87c1f1ec"} Jul 14 21:37:52.077294 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:52.077265 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:52.077933 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:52.077312 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:37:52.077933 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:52.077394 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 21:37:52.077933 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:52.077426 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cf444f6dc-f2bc5: secret "image-registry-tls" not found Jul 14 21:37:52.077933 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:52.077431 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 21:37:52.077933 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:52.077475 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls podName:4e461f5b-ff8f-4018-a3d2-9de160cabffd nodeName:}" failed. No retries permitted until 2026-07-14 21:37:56.077460904 +0000 UTC m=+42.616166554 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls") pod "dns-default-jrvlv" (UID: "4e461f5b-ff8f-4018-a3d2-9de160cabffd") : secret "dns-default-metrics-tls" not found Jul 14 21:37:52.077933 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:52.077489 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls podName:547803e8-dd34-48fb-bb45-f08182623df4 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:56.077482881 +0000 UTC m=+42.616188530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls") pod "image-registry-cf444f6dc-f2bc5" (UID: "547803e8-dd34-48fb-bb45-f08182623df4") : secret "image-registry-tls" not found Jul 14 21:37:52.183455 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:52.183419 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xxhmt" event={"ID":"9687cbd1-86bc-4429-9f4d-a8cda7399acb","Type":"ContainerStarted","Data":"b6e96bb7a5339d999721594988df3bc770d71f7fffec59ee88471d54b17e0730"} Jul 14 21:37:52.187088 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:52.187059 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2k4fl" event={"ID":"2a255c4c-c31b-4a1b-bd13-263359830473","Type":"ContainerStarted","Data":"0ccd8ffdf110a26425e55969993fc82b3032b100e70c493e5fccdb358e9fb66c"} Jul 14 21:37:52.205306 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:52.205251 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xxhmt" podStartSLOduration=1.8899601019999999 podStartE2EDuration="4.205229561s" totalImagesPullingTime="2.315269459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:37:52.204173317 +0000 UTC m=+38.742878988" watchObservedRunningTime="2026-07-14 21:37:52.205229561 +0000 UTC m=+38.743935235" Jul 14 21:37:52.233150 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:52.233097 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2k4fl" podStartSLOduration=8.049500148 podStartE2EDuration="38.233074946s" totalImagesPullingTime="30.183574798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:37:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:37:52.232239394 +0000 UTC m=+38.770945064" watchObservedRunningTime="2026-07-14 21:37:52.233074946 +0000 UTC m=+38.771780667" Jul 14 21:37:52.367107 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:52.367063 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:37:52.464903 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:52.464801 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:37:54.247382 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:54.247141 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:37:54.347050 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:54.347018 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:37:56.105944 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.105905 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:37:56.106328 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.105955 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:37:56.106328 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:56.106071 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 21:37:56.106328 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:56.106098 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cf444f6dc-f2bc5: secret "image-registry-tls" not found Jul 14 21:37:56.106328 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:56.106082 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 21:37:56.106328 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:56.106155 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls podName:4e461f5b-ff8f-4018-a3d2-9de160cabffd nodeName:}" failed. No retries permitted until 2026-07-14 21:38:04.106141905 +0000 UTC m=+50.644847568 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls") pod "dns-default-jrvlv" (UID: "4e461f5b-ff8f-4018-a3d2-9de160cabffd") : secret "dns-default-metrics-tls" not found Jul 14 21:37:56.106328 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:56.106169 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls podName:547803e8-dd34-48fb-bb45-f08182623df4 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:04.10616287 +0000 UTC m=+50.644868533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls") pod "image-registry-cf444f6dc-f2bc5" (UID: "547803e8-dd34-48fb-bb45-f08182623df4") : secret "image-registry-tls" not found Jul 14 21:37:56.578867 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.578790 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-8gw65"] Jul 14 21:37:56.613370 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.613333 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-8gw65"] Jul 14 21:37:56.613522 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.613451 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8gw65" Jul 14 21:37:56.616413 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.616393 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ksbhn\"" Jul 14 21:37:56.682095 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.682052 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm"] Jul 14 21:37:56.703428 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.703401 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-x65n5"] Jul 14 21:37:56.703606 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.703554 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" Jul 14 21:37:56.706836 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.706810 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 14 21:37:56.706954 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.706863 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fn6sz\"" Jul 14 21:37:56.707318 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.707298 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 14 21:37:56.707605 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.707576 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 14 21:37:56.710209 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.710165 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf2xg\" (UniqueName: \"kubernetes.io/projected/6c93b60b-5d37-4699-adc2-92d867cb37ac-kube-api-access-vf2xg\") pod \"network-check-source-869dcfcdcd-8gw65\" (UID: \"6c93b60b-5d37-4699-adc2-92d867cb37ac\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8gw65" Jul 14 21:37:56.723404 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.723375 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-cdl6v"] Jul 14 21:37:56.723571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.723499 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:37:56.726682 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.726652 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 14 21:37:56.726812 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.726796 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 14 21:37:56.726869 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.726826 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 14 21:37:56.726912 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.726880 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 14 21:37:56.727070 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.727058 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jshxk\"" Jul 14 21:37:56.733261 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.733244 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 14 21:37:56.749436 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.749407 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4"] Jul 14 21:37:56.749582 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.749542 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-cdl6v" Jul 14 21:37:56.752380 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.752359 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 14 21:37:56.752494 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.752359 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-pgzm2\"" Jul 14 21:37:56.752494 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.752441 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 14 21:37:56.767491 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.767472 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5c67f54db8-mlqfz"] Jul 14 21:37:56.767604 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.767587 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" Jul 14 21:37:56.772595 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.772572 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 14 21:37:56.772880 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.772853 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-k947l\"" Jul 14 21:37:56.773593 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.773575 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 21:37:56.774135 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.774114 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 21:37:56.774273 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.774258 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 14 21:37:56.794085 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.794056 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-cdl6v"] Jul 14 21:37:56.794085 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.794089 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm"] Jul 14 21:37:56.794251 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.794113 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-x65n5"] Jul 14 21:37:56.794251 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.794136 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c67f54db8-mlqfz"] Jul 14 21:37:56.794251 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.794150 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4"] Jul 14 21:37:56.794251 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.794175 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk"] Jul 14 21:37:56.794521 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.794273 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:56.798883 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.798864 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-sxltv\"" Jul 14 21:37:56.799329 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.799307 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 21:37:56.799671 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.799656 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 14 21:37:56.801101 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.800300 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 14 21:37:56.801101 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.800544 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 14 21:37:56.801315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.801298 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 14 21:37:56.808882 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.808863 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 21:37:56.811073 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.811054 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnzp4\" (UniqueName: \"kubernetes.io/projected/a6cb0783-1901-4441-a543-215ede6cef78-kube-api-access-hnzp4\") pod \"console-operator-7866497494-x65n5\" (UID: \"a6cb0783-1901-4441-a543-215ede6cef78\") " pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:37:56.811156 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.811079 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a6cb0783-1901-4441-a543-215ede6cef78-config\") pod \"console-operator-7866497494-x65n5\" (UID: \"a6cb0783-1901-4441-a543-215ede6cef78\") " pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:37:56.811156 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.811151 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf2xg\" (UniqueName: \"kubernetes.io/projected/6c93b60b-5d37-4699-adc2-92d867cb37ac-kube-api-access-vf2xg\") pod \"network-check-source-869dcfcdcd-8gw65\" (UID: \"6c93b60b-5d37-4699-adc2-92d867cb37ac\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8gw65" Jul 14 21:37:56.811253 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.811171 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-8xwjm\" (UID: \"be6e2b61-604a-43e4-a2e4-4379dbaf0b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" Jul 14 21:37:56.811253 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.811213 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4ngd\" (UniqueName: \"kubernetes.io/projected/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-kube-api-access-f4ngd\") pod \"cluster-samples-operator-65db858894-8xwjm\" (UID: \"be6e2b61-604a-43e4-a2e4-4379dbaf0b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" Jul 14 21:37:56.811253 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.811236 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a6cb0783-1901-4441-a543-215ede6cef78-serving-cert\") pod \"console-operator-7866497494-x65n5\" (UID: \"a6cb0783-1901-4441-a543-215ede6cef78\") " pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:37:56.811355 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.811259 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6cb0783-1901-4441-a543-215ede6cef78-trusted-ca\") pod \"console-operator-7866497494-x65n5\" (UID: \"a6cb0783-1901-4441-a543-215ede6cef78\") " pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:37:56.816271 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.816252 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-sgt5s"] Jul 14 21:37:56.816375 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.816360 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk" Jul 14 21:37:56.819848 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.819830 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 14 21:37:56.819938 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.819902 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jx85s\"" Jul 14 21:37:56.820015 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.819999 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 14 21:37:56.820083 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.820070 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 14 21:37:56.820645 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.820632 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 14 21:37:56.827609 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.827149 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf2xg\" (UniqueName: \"kubernetes.io/projected/6c93b60b-5d37-4699-adc2-92d867cb37ac-kube-api-access-vf2xg\") pod \"network-check-source-869dcfcdcd-8gw65\" (UID: \"6c93b60b-5d37-4699-adc2-92d867cb37ac\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8gw65" Jul 14 21:37:56.837887 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.837869 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-8cggq"] Jul 14 21:37:56.837986 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.837970 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:56.843910 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.843881 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 14 21:37:56.844634 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.844614 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-sncvp\"" Jul 14 21:37:56.844738 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.844708 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 21:37:56.845270 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.845248 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 14 21:37:56.846579 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.845647 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 21:37:56.849992 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.849970 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 14 21:37:56.862044 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.862026 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk"] Jul 14 21:37:56.862140 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.862055 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-sgt5s"] Jul 14 21:37:56.862140 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.862070 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-8cggq"] Jul 14 21:37:56.862229 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.862146 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-8cggq" Jul 14 21:37:56.871234 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.871213 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 14 21:37:56.871314 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.871298 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 14 21:37:56.876086 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.876072 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-z2lz9\"" Jul 14 21:37:56.912381 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.912340 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64"] Jul 14 21:37:56.912585 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.912563 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4ngd\" (UniqueName: \"kubernetes.io/projected/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-kube-api-access-f4ngd\") pod \"cluster-samples-operator-65db858894-8xwjm\" (UID: \"be6e2b61-604a-43e4-a2e4-4379dbaf0b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" Jul 14 21:37:56.912640 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.912599 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a6cb0783-1901-4441-a543-215ede6cef78-serving-cert\") pod \"console-operator-7866497494-x65n5\" (UID: \"a6cb0783-1901-4441-a543-215ede6cef78\") " pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:37:56.912689 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.912646 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d7kb\" (UniqueName: \"kubernetes.io/projected/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-kube-api-access-2d7kb\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:56.912689 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.912678 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-snapshots\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:56.912782 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.912711 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a6cb0783-1901-4441-a543-215ede6cef78-config\") pod \"console-operator-7866497494-x65n5\" (UID: \"a6cb0783-1901-4441-a543-215ede6cef78\") " pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:37:56.912782 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.912736 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-tmp\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:56.912906 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.912886 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmbn5\" (UniqueName: \"kubernetes.io/projected/56c0e606-2ba2-4a4e-9438-ea86a06152a9-kube-api-access-wmbn5\") pod \"service-ca-operator-7695fc6c8c-mm7vk\" (UID: \"56c0e606-2ba2-4a4e-9438-ea86a06152a9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk" Jul 14 21:37:56.912957 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.912914 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:56.912957 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.912940 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:56.913057 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.913026 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-default-certificate\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:56.913113 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.913066 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-8xwjm\" (UID: \"be6e2b61-604a-43e4-a2e4-4379dbaf0b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" Jul 14 21:37:56.913113 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.913095 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6cb0783-1901-4441-a543-215ede6cef78-trusted-ca\") pod \"console-operator-7866497494-x65n5\" (UID: \"a6cb0783-1901-4441-a543-215ede6cef78\") " pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:37:56.913214 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.913121 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/587d0871-09d8-4223-955c-52d205e0051b-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-82wc4\" (UID: \"587d0871-09d8-4223-955c-52d205e0051b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" Jul 14 21:37:56.913214 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:56.913151 2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 21:37:56.913364 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:56.913349 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls podName:be6e2b61-604a-43e4-a2e4-4379dbaf0b9c nodeName:}" failed. No retries permitted until 2026-07-14 21:37:57.413329129 +0000 UTC m=+43.952034796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls") pod "cluster-samples-operator-65db858894-8xwjm" (UID: "be6e2b61-604a-43e4-a2e4-4379dbaf0b9c") : secret "samples-operator-tls" not found Jul 14 21:37:56.913465 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.913157 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gj58\" (UniqueName: \"kubernetes.io/projected/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-kube-api-access-7gj58\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:56.913573 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.913487 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56c0e606-2ba2-4a4e-9438-ea86a06152a9-serving-cert\") pod \"service-ca-operator-7695fc6c8c-mm7vk\" (UID: \"56c0e606-2ba2-4a4e-9438-ea86a06152a9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk" Jul 14 21:37:56.913573 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.913527 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n58m\" (UniqueName: \"kubernetes.io/projected/b7676ddd-9ceb-4328-98a2-44172842a0d8-kube-api-access-7n58m\") pod \"volume-data-source-validator-78495597f6-cdl6v\" (UID: \"b7676ddd-9ceb-4328-98a2-44172842a0d8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-cdl6v" Jul 14 21:37:56.913573 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.913558 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvtdl\" (UniqueName: \"kubernetes.io/projected/587d0871-09d8-4223-955c-52d205e0051b-kube-api-access-jvtdl\") pod \"cluster-monitoring-operator-5f7f494dd4-82wc4\" (UID: \"587d0871-09d8-4223-955c-52d205e0051b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" Jul 14 21:37:56.913733 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.913578 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a6cb0783-1901-4441-a543-215ede6cef78-config\") pod \"console-operator-7866497494-x65n5\" (UID: \"a6cb0783-1901-4441-a543-215ede6cef78\") " pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:37:56.913733 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.913588 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnzp4\" (UniqueName: \"kubernetes.io/projected/a6cb0783-1901-4441-a543-215ede6cef78-kube-api-access-hnzp4\") pod \"console-operator-7866497494-x65n5\" (UID: \"a6cb0783-1901-4441-a543-215ede6cef78\") " pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:37:56.913733 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.913629 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-82wc4\" (UID: \"587d0871-09d8-4223-955c-52d205e0051b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" Jul 14 21:37:56.913733 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.913654 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56c0e606-2ba2-4a4e-9438-ea86a06152a9-config\") pod \"service-ca-operator-7695fc6c8c-mm7vk\" (UID: \"56c0e606-2ba2-4a4e-9438-ea86a06152a9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk" Jul 14 21:37:56.913733 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.913681 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:56.913733 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.913709 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:56.913989 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.913742 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-stats-auth\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:56.913989 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.913788 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-serving-cert\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:56.914149 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.914127 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6cb0783-1901-4441-a543-215ede6cef78-trusted-ca\") pod \"console-operator-7866497494-x65n5\" (UID: \"a6cb0783-1901-4441-a543-215ede6cef78\") " pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:37:56.924315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.924293 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8gw65" Jul 14 21:37:56.935623 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.935599 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a6cb0783-1901-4441-a543-215ede6cef78-serving-cert\") pod \"console-operator-7866497494-x65n5\" (UID: \"a6cb0783-1901-4441-a543-215ede6cef78\") " pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:37:56.938615 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.938446 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:37:56.941859 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.941407 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnzp4\" (UniqueName: \"kubernetes.io/projected/a6cb0783-1901-4441-a543-215ede6cef78-kube-api-access-hnzp4\") pod \"console-operator-7866497494-x65n5\" (UID: \"a6cb0783-1901-4441-a543-215ede6cef78\") " pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:37:56.941859 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.941807 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4ngd\" (UniqueName: \"kubernetes.io/projected/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-kube-api-access-f4ngd\") pod \"cluster-samples-operator-65db858894-8xwjm\" (UID: \"be6e2b61-604a-43e4-a2e4-4379dbaf0b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" Jul 14 21:37:56.942079 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.942058 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64" Jul 14 21:37:56.942794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.942771 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64"] Jul 14 21:37:56.952015 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.951989 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 14 21:37:56.952278 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.952262 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 14 21:37:56.956662 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.956636 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 14 21:37:56.956762 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.956688 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 14 21:37:56.956818 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:56.956783 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vvbw7\"" Jul 14 21:37:57.014432 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014398 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gj58\" (UniqueName: \"kubernetes.io/projected/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-kube-api-access-7gj58\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:57.014570 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014448 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56c0e606-2ba2-4a4e-9438-ea86a06152a9-serving-cert\") pod \"service-ca-operator-7695fc6c8c-mm7vk\" (UID: \"56c0e606-2ba2-4a4e-9438-ea86a06152a9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk" Jul 14 21:37:57.014570 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014503 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n58m\" (UniqueName: \"kubernetes.io/projected/b7676ddd-9ceb-4328-98a2-44172842a0d8-kube-api-access-7n58m\") pod \"volume-data-source-validator-78495597f6-cdl6v\" (UID: \"b7676ddd-9ceb-4328-98a2-44172842a0d8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-cdl6v" Jul 14 21:37:57.014570 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014533 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvtdl\" (UniqueName: \"kubernetes.io/projected/587d0871-09d8-4223-955c-52d205e0051b-kube-api-access-jvtdl\") pod \"cluster-monitoring-operator-5f7f494dd4-82wc4\" (UID: \"587d0871-09d8-4223-955c-52d205e0051b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" Jul 14 21:37:57.014662 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014568 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8cggq\" (UID: \"c03004c3-29b8-4813-85e1-5df04bd12a34\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8cggq" Jul 14 21:37:57.014662 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014615 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-77p64\" (UID: \"c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64" Jul 14 21:37:57.014662 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014644 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-82wc4\" (UID: \"587d0871-09d8-4223-955c-52d205e0051b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" Jul 14 21:37:57.014805 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014668 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56c0e606-2ba2-4a4e-9438-ea86a06152a9-config\") pod \"service-ca-operator-7695fc6c8c-mm7vk\" (UID: \"56c0e606-2ba2-4a4e-9438-ea86a06152a9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk" Jul 14 21:37:57.014805 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014692 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:57.014805 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014721 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:57.014805 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014748 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-stats-auth\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:57.014805 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014783 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-serving-cert\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:57.015040 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014818 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-77p64\" (UID: \"c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64" Jul 14 21:37:57.015040 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014848 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c03004c3-29b8-4813-85e1-5df04bd12a34-nginx-conf\") pod \"networking-console-plugin-6cf856758c-8cggq\" (UID: \"c03004c3-29b8-4813-85e1-5df04bd12a34\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8cggq" Jul 14 21:37:57.015040 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014889 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d7kb\" (UniqueName: \"kubernetes.io/projected/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-kube-api-access-2d7kb\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:57.015040 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014914 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-snapshots\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:57.015040 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014946 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct4jt\" (UniqueName: \"kubernetes.io/projected/c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3-kube-api-access-ct4jt\") pod \"kube-storage-version-migrator-operator-799cb7f958-77p64\" (UID: \"c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64" Jul 14 21:37:57.015040 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.014974 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-tmp\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:57.015040 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.015010 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmbn5\" (UniqueName: \"kubernetes.io/projected/56c0e606-2ba2-4a4e-9438-ea86a06152a9-kube-api-access-wmbn5\") pod \"service-ca-operator-7695fc6c8c-mm7vk\" (UID: \"56c0e606-2ba2-4a4e-9438-ea86a06152a9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk" Jul 14 21:37:57.015040 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.015033 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:57.015631 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.015058 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:57.015631 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.015105 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-default-certificate\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:57.015631 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.015141 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/587d0871-09d8-4223-955c-52d205e0051b-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-82wc4\" (UID: \"587d0871-09d8-4223-955c-52d205e0051b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" Jul 14 21:37:57.017165 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.016211 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/587d0871-09d8-4223-955c-52d205e0051b-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-82wc4\" (UID: \"587d0871-09d8-4223-955c-52d205e0051b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" Jul 14 21:37:57.017165 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:57.016232 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 21:37:57.017165 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:57.016308 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls podName:587d0871-09d8-4223-955c-52d205e0051b nodeName:}" failed. No retries permitted until 2026-07-14 21:37:57.516288558 +0000 UTC m=+44.054994224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-82wc4" (UID: "587d0871-09d8-4223-955c-52d205e0051b") : secret "cluster-monitoring-operator-tls" not found Jul 14 21:37:57.017165 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:57.016547 2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 21:37:57.017165 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:57.016593 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs podName:b1ed9b70-c79d-44a2-950c-af1d0b6bdd57 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:57.516577169 +0000 UTC m=+44.055282820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs") pod "router-default-5c67f54db8-mlqfz" (UID: "b1ed9b70-c79d-44a2-950c-af1d0b6bdd57") : secret "router-metrics-certs-default" not found Jul 14 21:37:57.017165 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.016736 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56c0e606-2ba2-4a4e-9438-ea86a06152a9-config\") pod \"service-ca-operator-7695fc6c8c-mm7vk\" (UID: \"56c0e606-2ba2-4a4e-9438-ea86a06152a9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk" Jul 14 21:37:57.017165 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.016978 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-snapshots\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:57.017165 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.017154 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:57.017653 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:57.017257 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle podName:b1ed9b70-c79d-44a2-950c-af1d0b6bdd57 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:57.517228826 +0000 UTC m=+44.055934489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle") pod "router-default-5c67f54db8-mlqfz" (UID: "b1ed9b70-c79d-44a2-950c-af1d0b6bdd57") : configmap references non-existent config key: service-ca.crt Jul 14 21:37:57.017653 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.017443 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-tmp\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:57.018249 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.018220 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:57.018645 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.018623 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-serving-cert\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:57.018950 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.018932 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56c0e606-2ba2-4a4e-9438-ea86a06152a9-serving-cert\") pod \"service-ca-operator-7695fc6c8c-mm7vk\" (UID: \"56c0e606-2ba2-4a4e-9438-ea86a06152a9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk" Jul 14 21:37:57.018993 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.018949 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-stats-auth\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:57.019890 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.019859 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-default-certificate\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:57.033031 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.032834 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:37:57.044324 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.044295 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gj58\" (UniqueName: \"kubernetes.io/projected/eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87-kube-api-access-7gj58\") pod \"insights-operator-68d9dd94cf-sgt5s\" (UID: \"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87\") " pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:57.047480 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.047456 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d7kb\" (UniqueName: \"kubernetes.io/projected/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-kube-api-access-2d7kb\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:57.048148 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.048130 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvtdl\" (UniqueName: \"kubernetes.io/projected/587d0871-09d8-4223-955c-52d205e0051b-kube-api-access-jvtdl\") pod \"cluster-monitoring-operator-5f7f494dd4-82wc4\" (UID: \"587d0871-09d8-4223-955c-52d205e0051b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" Jul 14 21:37:57.048275 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.048255 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n58m\" (UniqueName: \"kubernetes.io/projected/b7676ddd-9ceb-4328-98a2-44172842a0d8-kube-api-access-7n58m\") pod \"volume-data-source-validator-78495597f6-cdl6v\" (UID: \"b7676ddd-9ceb-4328-98a2-44172842a0d8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-cdl6v" Jul 14 21:37:57.050485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.050437 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:37:57.050688 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.050671 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmbn5\" (UniqueName: \"kubernetes.io/projected/56c0e606-2ba2-4a4e-9438-ea86a06152a9-kube-api-access-wmbn5\") pod \"service-ca-operator-7695fc6c8c-mm7vk\" (UID: \"56c0e606-2ba2-4a4e-9438-ea86a06152a9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk" Jul 14 21:37:57.057469 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.057449 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-cdl6v" Jul 14 21:37:57.104212 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.104143 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-8gw65"] Jul 14 21:37:57.106705 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:57.106658 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c93b60b_5d37_4699_adc2_92d867cb37ac.slice/crio-0f38366be6888ebc08123e4f448b492e32e8955e6397fa52e6b3910790c5396f WatchSource:0}: Error finding container 0f38366be6888ebc08123e4f448b492e32e8955e6397fa52e6b3910790c5396f: Status 404 returned error can't find the container with id 0f38366be6888ebc08123e4f448b492e32e8955e6397fa52e6b3910790c5396f Jul 14 21:37:57.116470 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.116443 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8cggq\" (UID: \"c03004c3-29b8-4813-85e1-5df04bd12a34\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8cggq" Jul 14 21:37:57.116610 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.116481 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-77p64\" (UID: \"c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64" Jul 14 21:37:57.116610 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.116523 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-77p64\" (UID: \"c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64" Jul 14 21:37:57.116610 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.116566 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c03004c3-29b8-4813-85e1-5df04bd12a34-nginx-conf\") pod \"networking-console-plugin-6cf856758c-8cggq\" (UID: \"c03004c3-29b8-4813-85e1-5df04bd12a34\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8cggq" Jul 14 21:37:57.116785 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.116613 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct4jt\" (UniqueName: \"kubernetes.io/projected/c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3-kube-api-access-ct4jt\") pod \"kube-storage-version-migrator-operator-799cb7f958-77p64\" (UID: \"c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64" Jul 14 21:37:57.117402 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:57.116961 2552 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 21:37:57.117402 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:57.117034 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert podName:c03004c3-29b8-4813-85e1-5df04bd12a34 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:57.617012053 +0000 UTC m=+44.155717724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-8cggq" (UID: "c03004c3-29b8-4813-85e1-5df04bd12a34") : secret "networking-console-plugin-cert" not found Jul 14 21:37:57.117571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.117415 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-77p64\" (UID: \"c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64" Jul 14 21:37:57.117707 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.117683 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c03004c3-29b8-4813-85e1-5df04bd12a34-nginx-conf\") pod \"networking-console-plugin-6cf856758c-8cggq\" (UID: \"c03004c3-29b8-4813-85e1-5df04bd12a34\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8cggq" Jul 14 21:37:57.119334 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.119283 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-77p64\" (UID: \"c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64" Jul 14 21:37:57.124779 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.124718 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk" Jul 14 21:37:57.132000 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.131739 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct4jt\" (UniqueName: \"kubernetes.io/projected/c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3-kube-api-access-ct4jt\") pod \"kube-storage-version-migrator-operator-799cb7f958-77p64\" (UID: \"c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64" Jul 14 21:37:57.147543 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.147519 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" Jul 14 21:37:57.152372 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.152345 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:37:57.192693 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.192650 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-x65n5"] Jul 14 21:37:57.198397 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:57.198358 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6cb0783_1901_4441_a543_215ede6cef78.slice/crio-19d4de3822be9672af751419030b31b6c29aa1d1f8a43b360f9e61ab4492f371 WatchSource:0}: Error finding container 19d4de3822be9672af751419030b31b6c29aa1d1f8a43b360f9e61ab4492f371: Status 404 returned error can't find the container with id 19d4de3822be9672af751419030b31b6c29aa1d1f8a43b360f9e61ab4492f371 Jul 14 21:37:57.204860 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.204826 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8gw65" event={"ID":"6c93b60b-5d37-4699-adc2-92d867cb37ac","Type":"ContainerStarted","Data":"0f38366be6888ebc08123e4f448b492e32e8955e6397fa52e6b3910790c5396f"} Jul 14 21:37:57.241029 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.240953 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-cdl6v"] Jul 14 21:37:57.243220 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.243079 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:37:57.245675 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:57.245628 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7676ddd_9ceb_4328_98a2_44172842a0d8.slice/crio-3bc22c10b8f69d1498cc26285f69c03d1fe7f09046370b7651c45bee5de785a7 WatchSource:0}: Error finding container 3bc22c10b8f69d1498cc26285f69c03d1fe7f09046370b7651c45bee5de785a7: Status 404 returned error can't find the container with id 3bc22c10b8f69d1498cc26285f69c03d1fe7f09046370b7651c45bee5de785a7 Jul 14 21:37:57.257702 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.257681 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64" Jul 14 21:37:57.301986 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.301953 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk"] Jul 14 21:37:57.304951 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:57.304927 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c0e606_2ba2_4a4e_9438_ea86a06152a9.slice/crio-39a9f73ab8214f945052dd927a2f14cae617d9e7146f9b7460a2479a52593102 WatchSource:0}: Error finding container 39a9f73ab8214f945052dd927a2f14cae617d9e7146f9b7460a2479a52593102: Status 404 returned error can't find the container with id 39a9f73ab8214f945052dd927a2f14cae617d9e7146f9b7460a2479a52593102 Jul 14 21:37:57.309041 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.309014 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:37:57.330991 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.330953 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-sgt5s"] Jul 14 21:37:57.335292 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:57.335258 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1713bc_74d9_4b7f_9d0e_c3450d0c6c87.slice/crio-17dadc6265420af8e9adf0807664ebb0f087f69b50bc88d67cc886398a9f9250 WatchSource:0}: Error finding container 17dadc6265420af8e9adf0807664ebb0f087f69b50bc88d67cc886398a9f9250: Status 404 returned error can't find the container with id 17dadc6265420af8e9adf0807664ebb0f087f69b50bc88d67cc886398a9f9250 Jul 14 21:37:57.419460 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.419428 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-8xwjm\" (UID: \"be6e2b61-604a-43e4-a2e4-4379dbaf0b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" Jul 14 21:37:57.419631 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:57.419608 2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 21:37:57.419713 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:57.419692 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls podName:be6e2b61-604a-43e4-a2e4-4379dbaf0b9c nodeName:}" failed. No retries permitted until 2026-07-14 21:37:58.419669015 +0000 UTC m=+44.958374683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls") pod "cluster-samples-operator-65db858894-8xwjm" (UID: "be6e2b61-604a-43e4-a2e4-4379dbaf0b9c") : secret "samples-operator-tls" not found Jul 14 21:37:57.431135 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.431101 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64"] Jul 14 21:37:57.434012 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:37:57.433989 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ca6103_e1eb_4d72_9bcc_444d8f8cedc3.slice/crio-dda364558dc56b4776acd277b1251e57355917cd1748c5aeb4a35f061e5f7d1e WatchSource:0}: Error finding container dda364558dc56b4776acd277b1251e57355917cd1748c5aeb4a35f061e5f7d1e: Status 404 returned error can't find the container with id dda364558dc56b4776acd277b1251e57355917cd1748c5aeb4a35f061e5f7d1e Jul 14 21:37:57.521029 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.520991 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-82wc4\" (UID: \"587d0871-09d8-4223-955c-52d205e0051b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" Jul 14 21:37:57.521224 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.521046 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:57.521224 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:57.521123 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 21:37:57.521224 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.521152 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:57.521224 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:57.521219 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle podName:b1ed9b70-c79d-44a2-950c-af1d0b6bdd57 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:58.521174096 +0000 UTC m=+45.059879749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle") pod "router-default-5c67f54db8-mlqfz" (UID: "b1ed9b70-c79d-44a2-950c-af1d0b6bdd57") : configmap references non-existent config key: service-ca.crt Jul 14 21:37:57.521445 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:57.521246 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls podName:587d0871-09d8-4223-955c-52d205e0051b nodeName:}" failed. No retries permitted until 2026-07-14 21:37:58.521235437 +0000 UTC m=+45.059941090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-82wc4" (UID: "587d0871-09d8-4223-955c-52d205e0051b") : secret "cluster-monitoring-operator-tls" not found Jul 14 21:37:57.521445 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:57.521264 2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 21:37:57.521445 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:57.521316 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs podName:b1ed9b70-c79d-44a2-950c-af1d0b6bdd57 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:58.521294986 +0000 UTC m=+45.060000653 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs") pod "router-default-5c67f54db8-mlqfz" (UID: "b1ed9b70-c79d-44a2-950c-af1d0b6bdd57") : secret "router-metrics-certs-default" not found Jul 14 21:37:57.622709 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:57.622621 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8cggq\" (UID: \"c03004c3-29b8-4813-85e1-5df04bd12a34\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8cggq" Jul 14 21:37:57.622872 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:57.622772 2552 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 21:37:57.622872 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:57.622856 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert podName:c03004c3-29b8-4813-85e1-5df04bd12a34 nodeName:}" failed. No retries permitted until 2026-07-14 21:37:58.622835039 +0000 UTC m=+45.161540708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-8cggq" (UID: "c03004c3-29b8-4813-85e1-5df04bd12a34") : secret "networking-console-plugin-cert" not found Jul 14 21:37:58.210667 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:58.210537 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk" event={"ID":"56c0e606-2ba2-4a4e-9438-ea86a06152a9","Type":"ContainerStarted","Data":"39a9f73ab8214f945052dd927a2f14cae617d9e7146f9b7460a2479a52593102"} Jul 14 21:37:58.216873 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:58.216836 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-cdl6v" event={"ID":"b7676ddd-9ceb-4328-98a2-44172842a0d8","Type":"ContainerStarted","Data":"3bc22c10b8f69d1498cc26285f69c03d1fe7f09046370b7651c45bee5de785a7"} Jul 14 21:37:58.219560 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:58.219433 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-x65n5" event={"ID":"a6cb0783-1901-4441-a543-215ede6cef78","Type":"ContainerStarted","Data":"19d4de3822be9672af751419030b31b6c29aa1d1f8a43b360f9e61ab4492f371"} Jul 14 21:37:58.221396 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:58.221329 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64" event={"ID":"c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3","Type":"ContainerStarted","Data":"dda364558dc56b4776acd277b1251e57355917cd1748c5aeb4a35f061e5f7d1e"} Jul 14 21:37:58.222688 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:58.222605 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" event={"ID":"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87","Type":"ContainerStarted","Data":"17dadc6265420af8e9adf0807664ebb0f087f69b50bc88d67cc886398a9f9250"} Jul 14 21:37:58.433794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:58.432961 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-8xwjm\" (UID: \"be6e2b61-604a-43e4-a2e4-4379dbaf0b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" Jul 14 21:37:58.433794 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:58.433245 2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 21:37:58.433794 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:58.433317 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls podName:be6e2b61-604a-43e4-a2e4-4379dbaf0b9c nodeName:}" failed. No retries permitted until 2026-07-14 21:38:00.43329804 +0000 UTC m=+46.972003919 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls") pod "cluster-samples-operator-65db858894-8xwjm" (UID: "be6e2b61-604a-43e4-a2e4-4379dbaf0b9c") : secret "samples-operator-tls" not found Jul 14 21:37:58.535134 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:58.534104 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-82wc4\" (UID: \"587d0871-09d8-4223-955c-52d205e0051b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" Jul 14 21:37:58.535134 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:58.534162 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:58.535134 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:58.534276 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:37:58.535134 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:58.534410 2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 21:37:58.535134 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:58.534477 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs podName:b1ed9b70-c79d-44a2-950c-af1d0b6bdd57 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:00.534455591 +0000 UTC m=+47.073161255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs") pod "router-default-5c67f54db8-mlqfz" (UID: "b1ed9b70-c79d-44a2-950c-af1d0b6bdd57") : secret "router-metrics-certs-default" not found Jul 14 21:37:58.535134 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:58.534873 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 21:37:58.535134 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:58.534923 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls podName:587d0871-09d8-4223-955c-52d205e0051b nodeName:}" failed. No retries permitted until 2026-07-14 21:38:00.534907961 +0000 UTC m=+47.073613626 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-82wc4" (UID: "587d0871-09d8-4223-955c-52d205e0051b") : secret "cluster-monitoring-operator-tls" not found Jul 14 21:37:58.535134 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:58.534989 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle podName:b1ed9b70-c79d-44a2-950c-af1d0b6bdd57 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:00.53497922 +0000 UTC m=+47.073684884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle") pod "router-default-5c67f54db8-mlqfz" (UID: "b1ed9b70-c79d-44a2-950c-af1d0b6bdd57") : configmap references non-existent config key: service-ca.crt Jul 14 21:37:58.636386 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:37:58.635705 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8cggq\" (UID: \"c03004c3-29b8-4813-85e1-5df04bd12a34\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8cggq" Jul 14 21:37:58.636386 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:58.635938 2552 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 21:37:58.636386 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:37:58.636009 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert podName:c03004c3-29b8-4813-85e1-5df04bd12a34 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:00.635988677 +0000 UTC m=+47.174694341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-8cggq" (UID: "c03004c3-29b8-4813-85e1-5df04bd12a34") : secret "networking-console-plugin-cert" not found Jul 14 21:38:00.451164 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:00.451124 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-8xwjm\" (UID: \"be6e2b61-604a-43e4-a2e4-4379dbaf0b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" Jul 14 21:38:00.451566 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:00.451310 2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 21:38:00.451566 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:00.451380 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls podName:be6e2b61-604a-43e4-a2e4-4379dbaf0b9c nodeName:}" failed. No retries permitted until 2026-07-14 21:38:04.451364173 +0000 UTC m=+50.990069827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls") pod "cluster-samples-operator-65db858894-8xwjm" (UID: "be6e2b61-604a-43e4-a2e4-4379dbaf0b9c") : secret "samples-operator-tls" not found Jul 14 21:38:00.552100 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:00.552062 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:38:00.552272 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:00.552168 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-82wc4\" (UID: \"587d0871-09d8-4223-955c-52d205e0051b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" Jul 14 21:38:00.552272 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:00.552201 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:38:00.552272 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:00.552217 2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 21:38:00.552419 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:00.552276 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 21:38:00.552419 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:00.552302 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs podName:b1ed9b70-c79d-44a2-950c-af1d0b6bdd57 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:04.552285944 +0000 UTC m=+51.090991593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs") pod "router-default-5c67f54db8-mlqfz" (UID: "b1ed9b70-c79d-44a2-950c-af1d0b6bdd57") : secret "router-metrics-certs-default" not found Jul 14 21:38:00.552419 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:00.552330 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls podName:587d0871-09d8-4223-955c-52d205e0051b nodeName:}" failed. No retries permitted until 2026-07-14 21:38:04.55231766 +0000 UTC m=+51.091023310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-82wc4" (UID: "587d0871-09d8-4223-955c-52d205e0051b") : secret "cluster-monitoring-operator-tls" not found Jul 14 21:38:00.552419 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:00.552346 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle podName:b1ed9b70-c79d-44a2-950c-af1d0b6bdd57 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:04.552338871 +0000 UTC m=+51.091044521 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle") pod "router-default-5c67f54db8-mlqfz" (UID: "b1ed9b70-c79d-44a2-950c-af1d0b6bdd57") : configmap references non-existent config key: service-ca.crt Jul 14 21:38:00.652875 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:00.652833 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8cggq\" (UID: \"c03004c3-29b8-4813-85e1-5df04bd12a34\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8cggq" Jul 14 21:38:00.653065 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:00.653008 2552 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 21:38:00.653132 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:00.653090 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert podName:c03004c3-29b8-4813-85e1-5df04bd12a34 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:04.653067654 +0000 UTC m=+51.191773304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-8cggq" (UID: "c03004c3-29b8-4813-85e1-5df04bd12a34") : secret "networking-console-plugin-cert" not found Jul 14 21:38:00.788641 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:00.788557 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:00.883417 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:00.883378 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:04.074975 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:04.074939 2552 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6cb0783_1901_4441_a543_215ede6cef78.slice/crio-90cc0e0f7208c156e2025926e562745e6620ea68ce1848490ba7cc4cce13fef4.scope\": RecentStats: unable to find data in memory cache]" Jul 14 21:38:04.187706 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.187664 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:38:04.187927 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.187722 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:38:04.187927 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:04.187827 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 21:38:04.187927 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:04.187848 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cf444f6dc-f2bc5: secret "image-registry-tls" not found Jul 14 21:38:04.187927 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:04.187860 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 21:38:04.187927 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:04.187912 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls podName:547803e8-dd34-48fb-bb45-f08182623df4 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:20.187895471 +0000 UTC m=+66.726601141 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls") pod "image-registry-cf444f6dc-f2bc5" (UID: "547803e8-dd34-48fb-bb45-f08182623df4") : secret "image-registry-tls" not found Jul 14 21:38:04.187927 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:04.187926 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls podName:4e461f5b-ff8f-4018-a3d2-9de160cabffd nodeName:}" failed. No retries permitted until 2026-07-14 21:38:20.187921091 +0000 UTC m=+66.726626740 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls") pod "dns-default-jrvlv" (UID: "4e461f5b-ff8f-4018-a3d2-9de160cabffd") : secret "dns-default-metrics-tls" not found Jul 14 21:38:04.246218 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.246164 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64" event={"ID":"c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3","Type":"ContainerStarted","Data":"cf92dc9d3dc033fb0691269fb1ba03a0df9c65e9ac54990d966df6fc102d61d2"} Jul 14 21:38:04.247943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.247917 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" event={"ID":"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87","Type":"ContainerStarted","Data":"327d484008c2f5f91962532d635820f1e8c608c9d4fa215f40ba0e98db11ba69"} Jul 14 21:38:04.249650 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.249615 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8gw65" event={"ID":"6c93b60b-5d37-4699-adc2-92d867cb37ac","Type":"ContainerStarted","Data":"c5f531a36b0df51682dde4afe0ce4ca630262c4dae529bbd05c07dae049a386a"} Jul 14 21:38:04.251424 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.251400 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk" event={"ID":"56c0e606-2ba2-4a4e-9438-ea86a06152a9","Type":"ContainerStarted","Data":"c1ac29fe81da688eb12a154867a41980f6012c15afa0b1bb914b9a7817e578db"} Jul 14 21:38:04.254537 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.253353 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-cdl6v" event={"ID":"b7676ddd-9ceb-4328-98a2-44172842a0d8","Type":"ContainerStarted","Data":"ec539e0d5ab7610ba8dd7fcf052232fce10d885c2a1b101225ed9d6b8fb489cd"} Jul 14 21:38:04.255366 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.255341 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/0.log" Jul 14 21:38:04.255618 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.255382 2552 generic.go:356] "Generic (PLEG): container finished" podID="a6cb0783-1901-4441-a543-215ede6cef78" containerID="90cc0e0f7208c156e2025926e562745e6620ea68ce1848490ba7cc4cce13fef4" exitCode=255 Jul 14 21:38:04.255618 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.255419 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-x65n5" event={"ID":"a6cb0783-1901-4441-a543-215ede6cef78","Type":"ContainerDied","Data":"90cc0e0f7208c156e2025926e562745e6620ea68ce1848490ba7cc4cce13fef4"} Jul 14 21:38:04.255747 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.255642 2552 scope.go:117] "RemoveContainer" containerID="90cc0e0f7208c156e2025926e562745e6620ea68ce1848490ba7cc4cce13fef4" Jul 14 21:38:04.356929 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.356870 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64" podStartSLOduration=2.022138467 podStartE2EDuration="8.356845853s" totalImagesPullingTime="6.334707386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:04.298632065 +0000 UTC m=+50.837337738" watchObservedRunningTime="2026-07-14 21:38:04.356845853 +0000 UTC m=+50.895551525" Jul 14 21:38:04.358310 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.358282 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:04.378201 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.378126 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk" podStartSLOduration=1.91473224 podStartE2EDuration="8.37810592s" totalImagesPullingTime="6.46337368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:04.377143325 +0000 UTC m=+50.915848997" watchObservedRunningTime="2026-07-14 21:38:04.37810592 +0000 UTC m=+50.916811573" Jul 14 21:38:04.409673 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.409584 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" podStartSLOduration=1.970630337 podStartE2EDuration="8.409564949s" totalImagesPullingTime="6.438934612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:04.408252007 +0000 UTC m=+50.946957680" watchObservedRunningTime="2026-07-14 21:38:04.409564949 +0000 UTC m=+50.948270623" Jul 14 21:38:04.439355 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.439286 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-cdl6v" podStartSLOduration=1.999704204 podStartE2EDuration="8.439264103s" totalImagesPullingTime="6.439559899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:04.438179867 +0000 UTC m=+50.976885549" watchObservedRunningTime="2026-07-14 21:38:04.439264103 +0000 UTC m=+50.977969776" Jul 14 21:38:04.446180 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.446143 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:04.460972 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.460864 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8gw65" podStartSLOduration=1.798898371 podStartE2EDuration="8.460844189s" totalImagesPullingTime="6.661945818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:04.460075372 +0000 UTC m=+50.998781044" watchObservedRunningTime="2026-07-14 21:38:04.460844189 +0000 UTC m=+50.999549862" Jul 14 21:38:04.497599 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.497555 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-8xwjm\" (UID: \"be6e2b61-604a-43e4-a2e4-4379dbaf0b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" Jul 14 21:38:04.497770 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:04.497739 2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 21:38:04.497843 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:04.497831 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls podName:be6e2b61-604a-43e4-a2e4-4379dbaf0b9c nodeName:}" failed. No retries permitted until 2026-07-14 21:38:12.497808168 +0000 UTC m=+59.036513844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls") pod "cluster-samples-operator-65db858894-8xwjm" (UID: "be6e2b61-604a-43e4-a2e4-4379dbaf0b9c") : secret "samples-operator-tls" not found Jul 14 21:38:04.556660 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.556561 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:04.598987 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.598948 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:38:04.599209 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.599080 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-82wc4\" (UID: \"587d0871-09d8-4223-955c-52d205e0051b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" Jul 14 21:38:04.599209 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.599117 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:38:04.599209 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:04.599128 2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 21:38:04.599368 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:04.599212 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs podName:b1ed9b70-c79d-44a2-950c-af1d0b6bdd57 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:12.599178787 +0000 UTC m=+59.137884460 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs") pod "router-default-5c67f54db8-mlqfz" (UID: "b1ed9b70-c79d-44a2-950c-af1d0b6bdd57") : secret "router-metrics-certs-default" not found Jul 14 21:38:04.599368 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:04.599276 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle podName:b1ed9b70-c79d-44a2-950c-af1d0b6bdd57 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:12.599260909 +0000 UTC m=+59.137966562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle") pod "router-default-5c67f54db8-mlqfz" (UID: "b1ed9b70-c79d-44a2-950c-af1d0b6bdd57") : configmap references non-existent config key: service-ca.crt Jul 14 21:38:04.599368 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:04.599291 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 21:38:04.599368 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:04.599364 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls podName:587d0871-09d8-4223-955c-52d205e0051b nodeName:}" failed. No retries permitted until 2026-07-14 21:38:12.599348219 +0000 UTC m=+59.138053884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-82wc4" (UID: "587d0871-09d8-4223-955c-52d205e0051b") : secret "cluster-monitoring-operator-tls" not found Jul 14 21:38:04.660584 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.660506 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:04.700521 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.700469 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8cggq\" (UID: \"c03004c3-29b8-4813-85e1-5df04bd12a34\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8cggq" Jul 14 21:38:04.700724 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:04.700640 2552 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 21:38:04.700784 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:04.700732 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert podName:c03004c3-29b8-4813-85e1-5df04bd12a34 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:12.700709636 +0000 UTC m=+59.239415302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-8cggq" (UID: "c03004c3-29b8-4813-85e1-5df04bd12a34") : secret "networking-console-plugin-cert" not found Jul 14 21:38:04.760134 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:04.760087 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:05.224789 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.224746 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:05.261773 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.261741 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 21:38:05.262243 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.262224 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/0.log" Jul 14 21:38:05.262338 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.262270 2552 generic.go:356] "Generic (PLEG): container finished" podID="a6cb0783-1901-4441-a543-215ede6cef78" containerID="e543cc19995ff8f7baebad90d0c30fb14c6425ff479d8953264918764ee832fa" exitCode=255 Jul 14 21:38:05.262412 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.262368 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-x65n5" event={"ID":"a6cb0783-1901-4441-a543-215ede6cef78","Type":"ContainerDied","Data":"e543cc19995ff8f7baebad90d0c30fb14c6425ff479d8953264918764ee832fa"} Jul 14 21:38:05.262490 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.262475 2552 scope.go:117] "RemoveContainer" containerID="90cc0e0f7208c156e2025926e562745e6620ea68ce1848490ba7cc4cce13fef4" Jul 14 21:38:05.262977 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.262960 2552 scope.go:117] "RemoveContainer" containerID="e543cc19995ff8f7baebad90d0c30fb14c6425ff479d8953264918764ee832fa" Jul 14 21:38:05.263285 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:05.263265 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-x65n5_openshift-console-operator(a6cb0783-1901-4441-a543-215ede6cef78)\"" pod="openshift-console-operator/console-operator-7866497494-x65n5" podUID="a6cb0783-1901-4441-a543-215ede6cef78" Jul 14 21:38:05.300860 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.300810 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:05.372854 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.372808 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:05.518327 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.518233 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:05.626208 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.626162 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:05.720503 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.720468 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:05.794306 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.794214 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:05.950665 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.950631 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pcscn"] Jul 14 21:38:05.982593 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.982555 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pcscn"] Jul 14 21:38:05.982763 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.982669 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:05.984246 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.984216 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:05.985865 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.985845 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 21:38:05.986369 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.986349 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 21:38:05.986369 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:05.986363 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-62q7v\"" Jul 14 21:38:06.013353 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.013319 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c40e5ea-9442-4e32-8636-999f09c6cee6-crio-socket\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:06.013353 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.013351 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:06.013545 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.013411 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:06.013580 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.013552 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c40e5ea-9442-4e32-8636-999f09c6cee6-data-volume\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:06.013617 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.013600 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wptnb\" (UniqueName: \"kubernetes.io/projected/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-api-access-wptnb\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:06.058290 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.058184 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:06.114289 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.114256 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c40e5ea-9442-4e32-8636-999f09c6cee6-data-volume\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:06.114450 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.114307 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wptnb\" (UniqueName: \"kubernetes.io/projected/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-api-access-wptnb\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:06.114450 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.114340 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c40e5ea-9442-4e32-8636-999f09c6cee6-crio-socket\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:06.114450 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.114359 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:06.114450 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.114385 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:06.114650 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:06.114449 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 21:38:06.114650 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:06.114512 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 21:38:06.114650 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:06.114552 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm podName:6c40e5ea-9442-4e32-8636-999f09c6cee6 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:06.614533655 +0000 UTC m=+53.153239305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pcscn" (UID: "6c40e5ea-9442-4e32-8636-999f09c6cee6") : configmap "kube-rbac-proxy" not found Jul 14 21:38:06.114650 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.114557 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c40e5ea-9442-4e32-8636-999f09c6cee6-crio-socket\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:06.114650 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:06.114572 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls podName:6c40e5ea-9442-4e32-8636-999f09c6cee6 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:06.614561914 +0000 UTC m=+53.153267570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pcscn" (UID: "6c40e5ea-9442-4e32-8636-999f09c6cee6") : secret "insights-runtime-extractor-tls" not found Jul 14 21:38:06.114650 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.114621 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c40e5ea-9442-4e32-8636-999f09c6cee6-data-volume\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:06.125202 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.125172 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wptnb\" (UniqueName: \"kubernetes.io/projected/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-api-access-wptnb\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:06.220870 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.220830 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:06.266522 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.266495 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 21:38:06.266885 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.266839 2552 scope.go:117] "RemoveContainer" containerID="e543cc19995ff8f7baebad90d0c30fb14c6425ff479d8953264918764ee832fa" Jul 14 21:38:06.267066 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:06.267046 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-x65n5_openshift-console-operator(a6cb0783-1901-4441-a543-215ede6cef78)\"" pod="openshift-console-operator/console-operator-7866497494-x65n5" podUID="a6cb0783-1901-4441-a543-215ede6cef78" Jul 14 21:38:06.295882 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.295848 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:06.618941 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.618900 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:06.619149 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:06.618949 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:06.619149 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:06.619073 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 21:38:06.619149 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:06.619102 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 21:38:06.619149 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:06.619134 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm podName:6c40e5ea-9442-4e32-8636-999f09c6cee6 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:07.619115748 +0000 UTC m=+54.157821419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pcscn" (UID: "6c40e5ea-9442-4e32-8636-999f09c6cee6") : configmap "kube-rbac-proxy" not found Jul 14 21:38:06.619345 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:06.619170 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls podName:6c40e5ea-9442-4e32-8636-999f09c6cee6 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:07.61915049 +0000 UTC m=+54.157856154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pcscn" (UID: "6c40e5ea-9442-4e32-8636-999f09c6cee6") : secret "insights-runtime-extractor-tls" not found Jul 14 21:38:07.033364 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:07.033267 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:38:07.033518 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:07.033373 2552 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:38:07.147108 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:07.147066 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:07.245554 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:07.245523 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:07.270169 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:07.270140 2552 scope.go:117] "RemoveContainer" containerID="e543cc19995ff8f7baebad90d0c30fb14c6425ff479d8953264918764ee832fa" Jul 14 21:38:07.270579 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:07.270328 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-x65n5_openshift-console-operator(a6cb0783-1901-4441-a543-215ede6cef78)\"" pod="openshift-console-operator/console-operator-7866497494-x65n5" podUID="a6cb0783-1901-4441-a543-215ede6cef78" Jul 14 21:38:07.628185 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:07.628139 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:07.628384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:07.628223 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:07.628384 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:07.628302 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 21:38:07.628384 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:07.628338 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 21:38:07.628384 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:07.628372 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls podName:6c40e5ea-9442-4e32-8636-999f09c6cee6 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:09.628353193 +0000 UTC m=+56.167058859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pcscn" (UID: "6c40e5ea-9442-4e32-8636-999f09c6cee6") : secret "insights-runtime-extractor-tls" not found Jul 14 21:38:07.628523 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:07.628396 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm podName:6c40e5ea-9442-4e32-8636-999f09c6cee6 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:09.628382877 +0000 UTC m=+56.167088528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pcscn" (UID: "6c40e5ea-9442-4e32-8636-999f09c6cee6") : configmap "kube-rbac-proxy" not found Jul 14 21:38:07.891151 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:07.891065 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:08.010062 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:08.010026 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:08.330356 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:08.330280 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c4vgt_ba716d4b-af3d-44a4-a7d0-d134e8c805f3/dns-node-resolver/0.log" Jul 14 21:38:09.124207 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:09.124172 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2545q_b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10/node-ca/0.log" Jul 14 21:38:09.262157 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:09.262112 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:09.647033 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:09.646998 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:09.647484 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:09.647045 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:09.647484 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:09.647135 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 21:38:09.647484 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:09.647183 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 21:38:09.647484 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:09.647203 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm podName:6c40e5ea-9442-4e32-8636-999f09c6cee6 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:13.647174549 +0000 UTC m=+60.185880199 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pcscn" (UID: "6c40e5ea-9442-4e32-8636-999f09c6cee6") : configmap "kube-rbac-proxy" not found Jul 14 21:38:09.647484 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:09.647277 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls podName:6c40e5ea-9442-4e32-8636-999f09c6cee6 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:13.647258674 +0000 UTC m=+60.185964328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pcscn" (UID: "6c40e5ea-9442-4e32-8636-999f09c6cee6") : secret "insights-runtime-extractor-tls" not found Jul 14 21:38:10.122911 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:10.122881 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xxhmt_9687cbd1-86bc-4429-9f4d-a8cda7399acb/serve-healthcheck-canary/0.log" Jul 14 21:38:10.753382 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:10.753350 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-77p64_c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3/kube-storage-version-migrator-operator/0.log" Jul 14 21:38:11.462097 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:11.462060 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:11.882944 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:11.882911 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:12.198772 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:12.198668 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:12.277665 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:12.277628 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:12.469109 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:12.469018 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:12.566254 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:12.566219 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:12.571328 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:12.571299 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-8xwjm\" (UID: \"be6e2b61-604a-43e4-a2e4-4379dbaf0b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" Jul 14 21:38:12.571476 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:12.571458 2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 21:38:12.571544 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:12.571535 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls podName:be6e2b61-604a-43e4-a2e4-4379dbaf0b9c nodeName:}" failed. No retries permitted until 2026-07-14 21:38:28.571520509 +0000 UTC m=+75.110226182 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls") pod "cluster-samples-operator-65db858894-8xwjm" (UID: "be6e2b61-604a-43e4-a2e4-4379dbaf0b9c") : secret "samples-operator-tls" not found Jul 14 21:38:12.671988 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:12.671949 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:38:12.672151 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:12.672075 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-82wc4\" (UID: \"587d0871-09d8-4223-955c-52d205e0051b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" Jul 14 21:38:12.672151 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:12.672110 2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 21:38:12.672264 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:12.672109 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:38:12.672264 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:12.672184 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs podName:b1ed9b70-c79d-44a2-950c-af1d0b6bdd57 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:28.672165129 +0000 UTC m=+75.210870798 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs") pod "router-default-5c67f54db8-mlqfz" (UID: "b1ed9b70-c79d-44a2-950c-af1d0b6bdd57") : secret "router-metrics-certs-default" not found Jul 14 21:38:12.672264 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:12.672223 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle podName:b1ed9b70-c79d-44a2-950c-af1d0b6bdd57 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:28.672213776 +0000 UTC m=+75.210919434 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle") pod "router-default-5c67f54db8-mlqfz" (UID: "b1ed9b70-c79d-44a2-950c-af1d0b6bdd57") : configmap references non-existent config key: service-ca.crt Jul 14 21:38:12.672264 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:12.672243 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 21:38:12.672420 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:12.672323 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls podName:587d0871-09d8-4223-955c-52d205e0051b nodeName:}" failed. No retries permitted until 2026-07-14 21:38:28.672305786 +0000 UTC m=+75.211011456 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-82wc4" (UID: "587d0871-09d8-4223-955c-52d205e0051b") : secret "cluster-monitoring-operator-tls" not found Jul 14 21:38:12.773594 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:12.773500 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8cggq\" (UID: \"c03004c3-29b8-4813-85e1-5df04bd12a34\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8cggq" Jul 14 21:38:12.773744 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:12.773660 2552 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 21:38:12.773744 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:12.773725 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert podName:c03004c3-29b8-4813-85e1-5df04bd12a34 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:28.773710545 +0000 UTC m=+75.312416209 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-8cggq" (UID: "c03004c3-29b8-4813-85e1-5df04bd12a34") : secret "networking-console-plugin-cert" not found Jul 14 21:38:12.777162 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:12.777131 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:12.863595 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:12.863521 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:13.366580 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:13.366547 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:13.682623 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:13.682538 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:13.682623 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:13.682587 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:13.682815 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:13.682654 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 21:38:13.682815 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:13.682693 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 21:38:13.682815 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:13.682769 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls podName:6c40e5ea-9442-4e32-8636-999f09c6cee6 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:21.682748313 +0000 UTC m=+68.221453987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pcscn" (UID: "6c40e5ea-9442-4e32-8636-999f09c6cee6") : secret "insights-runtime-extractor-tls" not found Jul 14 21:38:13.682815 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:38:13.682793 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm podName:6c40e5ea-9442-4e32-8636-999f09c6cee6 nodeName:}" failed. No retries permitted until 2026-07-14 21:38:21.682785804 +0000 UTC m=+68.221491454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pcscn" (UID: "6c40e5ea-9442-4e32-8636-999f09c6cee6") : configmap "kube-rbac-proxy" not found Jul 14 21:38:13.978079 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:13.977987 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:14.161907 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:14.161881 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxphv" Jul 14 21:38:14.173306 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:14.173270 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:14.585648 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:14.585606 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:14.765086 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:14.765056 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:15.367544 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:15.367512 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:15.563518 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:15.563488 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:15.774835 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:15.774743 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:15.874987 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:15.874950 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:15.976165 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:15.976131 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:16.097321 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:16.097282 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:16.369046 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:16.368950 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:16.426938 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:16.426901 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:17.188167 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:17.188134 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:17.589119 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:17.589080 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:17.769234 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:17.769180 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:17.967099 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:17.967016 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:18.212725 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:18.212688 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:18.370555 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:18.370513 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:18.762888 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:18.762795 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:18.970762 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:18.970732 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:19.835631 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:19.835587 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret\") pod \"global-pull-secret-syncer-f2b5s\" (UID: \"d0336426-165c-4987-beaa-f74fe715d480\") " pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:38:19.836102 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:19.835641 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs\") pod \"network-metrics-daemon-czmxj\" (UID: \"fab63e1e-b743-4ed6-bbbd-6686855f1a43\") " pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:38:19.836102 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:19.835676 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lffmx\" (UniqueName: \"kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx\") pod \"network-check-target-8hxtk\" (UID: \"8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692\") " pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:38:19.838246 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:19.838225 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lffmx\" (UniqueName: \"kubernetes.io/projected/8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692-kube-api-access-lffmx\") pod \"network-check-target-8hxtk\" (UID: \"8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692\") " pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:38:19.839876 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:19.839853 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 21:38:19.840665 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:19.840648 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 21:38:19.848092 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:19.848063 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fab63e1e-b743-4ed6-bbbd-6686855f1a43-metrics-certs\") pod \"network-metrics-daemon-czmxj\" (UID: \"fab63e1e-b743-4ed6-bbbd-6686855f1a43\") " pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:38:19.848092 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:19.848087 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0336426-165c-4987-beaa-f74fe715d480-original-pull-secret\") pod \"global-pull-secret-syncer-f2b5s\" (UID: \"d0336426-165c-4987-beaa-f74fe715d480\") " pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:38:19.900698 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:19.900657 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bkps7\"" Jul 14 21:38:19.905164 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:19.905141 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2b5s" Jul 14 21:38:19.908272 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:19.908256 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-czmxj" Jul 14 21:38:19.914420 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:19.914400 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2rjnp\"" Jul 14 21:38:19.921889 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:19.921869 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:38:20.082998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.082974 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f2b5s"] Jul 14 21:38:20.085070 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:38:20.085034 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0336426_165c_4987_beaa_f74fe715d480.slice/crio-7a616544efa93a71c3ee241074c8ff22a9babc20dedb25be2ece0a83e96a4bbc WatchSource:0}: Error finding container 7a616544efa93a71c3ee241074c8ff22a9babc20dedb25be2ece0a83e96a4bbc: Status 404 returned error can't find the container with id 7a616544efa93a71c3ee241074c8ff22a9babc20dedb25be2ece0a83e96a4bbc Jul 14 21:38:20.107071 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.107042 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-czmxj"] Jul 14 21:38:20.109575 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:38:20.109552 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab63e1e_b743_4ed6_bbbd_6686855f1a43.slice/crio-2ce697160c0876259945933f7348f17e15b6b74764de270034af51226c9202e2 WatchSource:0}: Error finding container 2ce697160c0876259945933f7348f17e15b6b74764de270034af51226c9202e2: Status 404 returned error can't find the container with id 2ce697160c0876259945933f7348f17e15b6b74764de270034af51226c9202e2 Jul 14 21:38:20.114296 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.114270 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8hxtk"] Jul 14 21:38:20.116723 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:38:20.116700 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0d1c3e_fe9e_476e_9dfa_5778d2e2b692.slice/crio-0adbd976b5458a6b14e6bfff3647d46038cc45cf7d13fb8c91c8055358dd5712 WatchSource:0}: Error finding container 0adbd976b5458a6b14e6bfff3647d46038cc45cf7d13fb8c91c8055358dd5712: Status 404 returned error can't find the container with id 0adbd976b5458a6b14e6bfff3647d46038cc45cf7d13fb8c91c8055358dd5712 Jul 14 21:38:20.117913 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.117886 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:20.176721 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.176694 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:20.239479 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.239451 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:38:20.239623 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.239500 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:38:20.241697 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.241666 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e461f5b-ff8f-4018-a3d2-9de160cabffd-metrics-tls\") pod \"dns-default-jrvlv\" (UID: \"4e461f5b-ff8f-4018-a3d2-9de160cabffd\") " pod="openshift-dns/dns-default-jrvlv" Jul 14 21:38:20.241864 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.241841 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls\") pod \"image-registry-cf444f6dc-f2bc5\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:38:20.308710 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.308666 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-czmxj" event={"ID":"fab63e1e-b743-4ed6-bbbd-6686855f1a43","Type":"ContainerStarted","Data":"2ce697160c0876259945933f7348f17e15b6b74764de270034af51226c9202e2"} Jul 14 21:38:20.309725 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.309697 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f2b5s" event={"ID":"d0336426-165c-4987-beaa-f74fe715d480","Type":"ContainerStarted","Data":"7a616544efa93a71c3ee241074c8ff22a9babc20dedb25be2ece0a83e96a4bbc"} Jul 14 21:38:20.311066 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.311045 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8hxtk" event={"ID":"8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692","Type":"ContainerStarted","Data":"bff2a08eb23b945469b3bcc7953ca340ec46416ce1f52871cd7b192cc33d5e82"} Jul 14 21:38:20.311172 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.311078 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8hxtk" event={"ID":"8f0d1c3e-fe9e-476e-9dfa-5778d2e2b692","Type":"ContainerStarted","Data":"0adbd976b5458a6b14e6bfff3647d46038cc45cf7d13fb8c91c8055358dd5712"} Jul 14 21:38:20.311245 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.311176 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:38:20.330284 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.330235 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8hxtk" podStartSLOduration=66.330215358 podStartE2EDuration="1m6.330215358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:20.328888235 +0000 UTC m=+66.867593908" watchObservedRunningTime="2026-07-14 21:38:20.330215358 +0000 UTC m=+66.868921024" Jul 14 21:38:20.365447 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.365417 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:20.437317 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.437284 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tcngh\"" Jul 14 21:38:20.444810 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.444776 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:38:20.490601 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.490566 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zdsh8\"" Jul 14 21:38:20.497952 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.497930 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jrvlv" Jul 14 21:38:20.568657 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.568609 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:20.616829 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.616750 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cf444f6dc-f2bc5"] Jul 14 21:38:20.624781 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:38:20.624742 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547803e8_dd34_48fb_bb45_f08182623df4.slice/crio-4f83ccf14e00d2d14bdb3068dc6aeff2db0e8566600d8fb43b240c7780bc0d3d WatchSource:0}: Error finding container 4f83ccf14e00d2d14bdb3068dc6aeff2db0e8566600d8fb43b240c7780bc0d3d: Status 404 returned error can't find the container with id 4f83ccf14e00d2d14bdb3068dc6aeff2db0e8566600d8fb43b240c7780bc0d3d Jul 14 21:38:20.681745 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.681715 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jrvlv"] Jul 14 21:38:20.684732 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:38:20.684699 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e461f5b_ff8f_4018_a3d2_9de160cabffd.slice/crio-2699fea0c13ae57217908189ebe4060c02e89159fecca57b9d13c98fb599184b WatchSource:0}: Error finding container 2699fea0c13ae57217908189ebe4060c02e89159fecca57b9d13c98fb599184b: Status 404 returned error can't find the container with id 2699fea0c13ae57217908189ebe4060c02e89159fecca57b9d13c98fb599184b Jul 14 21:38:20.775209 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.775151 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:20.963997 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:20.963903 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:21.317317 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:21.317228 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jrvlv" event={"ID":"4e461f5b-ff8f-4018-a3d2-9de160cabffd","Type":"ContainerStarted","Data":"2699fea0c13ae57217908189ebe4060c02e89159fecca57b9d13c98fb599184b"} Jul 14 21:38:21.319362 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:21.319328 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" event={"ID":"547803e8-dd34-48fb-bb45-f08182623df4","Type":"ContainerStarted","Data":"b2abe284fb1c80b3c0199a94472b109e74f4cd707638cca424d83168b356b357"} Jul 14 21:38:21.319509 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:21.319391 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" event={"ID":"547803e8-dd34-48fb-bb45-f08182623df4","Type":"ContainerStarted","Data":"4f83ccf14e00d2d14bdb3068dc6aeff2db0e8566600d8fb43b240c7780bc0d3d"} Jul 14 21:38:21.319618 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:21.319591 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:38:21.344774 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:21.344720 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" podStartSLOduration=67.344700859 podStartE2EDuration="1m7.344700859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:21.343968229 +0000 UTC m=+67.882673902" watchObservedRunningTime="2026-07-14 21:38:21.344700859 +0000 UTC m=+67.883406532" Jul 14 21:38:21.753864 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:21.753826 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:21.754034 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:21.753891 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:21.755379 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:21.754979 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c40e5ea-9442-4e32-8636-999f09c6cee6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:21.765367 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:21.765340 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c40e5ea-9442-4e32-8636-999f09c6cee6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pcscn\" (UID: \"6c40e5ea-9442-4e32-8636-999f09c6cee6\") " pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:21.894770 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:21.894731 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-62q7v\"" Jul 14 21:38:21.903003 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:21.902977 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pcscn" Jul 14 21:38:21.966107 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:21.966058 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:22.103637 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:22.103569 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pcscn"] Jul 14 21:38:22.105976 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:38:22.105944 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c40e5ea_9442_4e32_8636_999f09c6cee6.slice/crio-807c4a26270473d8035e2f3e9a9fc22e4726ffae4df2cd46c4253acd2d991c48 WatchSource:0}: Error finding container 807c4a26270473d8035e2f3e9a9fc22e4726ffae4df2cd46c4253acd2d991c48: Status 404 returned error can't find the container with id 807c4a26270473d8035e2f3e9a9fc22e4726ffae4df2cd46c4253acd2d991c48 Jul 14 21:38:22.168302 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:22.168265 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:22.331269 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:22.331230 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pcscn" event={"ID":"6c40e5ea-9442-4e32-8636-999f09c6cee6","Type":"ContainerStarted","Data":"807c4a26270473d8035e2f3e9a9fc22e4726ffae4df2cd46c4253acd2d991c48"} Jul 14 21:38:22.333437 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:22.333409 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-czmxj" event={"ID":"fab63e1e-b743-4ed6-bbbd-6686855f1a43","Type":"ContainerStarted","Data":"23809f9c2314c8cacd48f1a6214dcc63bd62db1797b1791ca37bb961c06b8213"} Jul 14 21:38:22.333588 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:22.333465 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-czmxj" event={"ID":"fab63e1e-b743-4ed6-bbbd-6686855f1a43","Type":"ContainerStarted","Data":"3e8f5bbcef72688277e786ec1268a582b43ebc2cc2f77e0e9af2f0cd61256d3c"} Jul 14 21:38:22.357439 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:22.357324 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-czmxj" podStartSLOduration=66.934137317 podStartE2EDuration="1m8.357307059s" totalImagesPullingTime="1.423169742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:22.356971616 +0000 UTC m=+68.895677303" watchObservedRunningTime="2026-07-14 21:38:22.357307059 +0000 UTC m=+68.896012734" Jul 14 21:38:22.365176 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:22.365128 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:22.568151 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:22.568110 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:22.986966 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:22.986938 2552 scope.go:117] "RemoveContainer" containerID="e543cc19995ff8f7baebad90d0c30fb14c6425ff479d8953264918764ee832fa" Jul 14 21:38:25.348286 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:25.348233 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jrvlv" event={"ID":"4e461f5b-ff8f-4018-a3d2-9de160cabffd","Type":"ContainerStarted","Data":"e867979a5d4a559602afbb58bb51953ccea256ea2cd7c0de6d3030c66e829321"} Jul 14 21:38:25.348810 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:25.348375 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jrvlv" event={"ID":"4e461f5b-ff8f-4018-a3d2-9de160cabffd","Type":"ContainerStarted","Data":"971900d61d6eed0c3c5cfabd9c16254fd3c7654b13dae582bc1799641e56abe3"} Jul 14 21:38:25.348810 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:25.348511 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jrvlv" Jul 14 21:38:25.350121 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:25.350073 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f2b5s" event={"ID":"d0336426-165c-4987-beaa-f74fe715d480","Type":"ContainerStarted","Data":"598ebfa89f16aefd4768ef3e937da00d689c88c5b22ba1fc2b463084c3c3395e"} Jul 14 21:38:25.352241 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:25.352218 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 21:38:25.352350 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:25.352267 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-x65n5" event={"ID":"a6cb0783-1901-4441-a543-215ede6cef78","Type":"ContainerStarted","Data":"a9f42d4f284ea4f707f58fe21b2d585f5f000043793f502344b9b8ad3f2495da"} Jul 14 21:38:25.352680 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:25.352659 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:38:25.368874 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:25.368820 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jrvlv" podStartSLOduration=33.731031252 podStartE2EDuration="37.368799297s" totalImagesPullingTime="3.637768045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:25.36820785 +0000 UTC m=+71.906913515" watchObservedRunningTime="2026-07-14 21:38:25.368799297 +0000 UTC m=+71.907504972" Jul 14 21:38:25.388039 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:25.387983 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-x65n5" podStartSLOduration=22.819714076 podStartE2EDuration="29.387966436s" totalImagesPullingTime="6.56825236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:25.387453139 +0000 UTC m=+71.926158834" watchObservedRunningTime="2026-07-14 21:38:25.387966436 +0000 UTC m=+71.926672107" Jul 14 21:38:25.405568 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:25.405508 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f2b5s" podStartSLOduration=67.157820009 podStartE2EDuration="1m11.405486193s" totalImagesPullingTime="4.247666184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:25.404836336 +0000 UTC m=+71.943542008" watchObservedRunningTime="2026-07-14 21:38:25.405486193 +0000 UTC m=+71.944191866" Jul 14 21:38:25.667876 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:25.667784 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-x65n5" Jul 14 21:38:25.879239 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:25.879202 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:25.975080 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:25.974980 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:26.381460 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:26.381335 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:26.537439 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:26.537403 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:26.730082 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:26.729989 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:26.831149 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:26.831115 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:26.941404 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:26.941368 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:27.010569 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:27.010470 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:27.346136 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:27.346098 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:27.545226 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:27.545167 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:28.325203 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.325148 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r"] Jul 14 21:38:28.328644 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.328618 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r" Jul 14 21:38:28.334520 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.334491 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 14 21:38:28.336121 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.336099 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 14 21:38:28.336915 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.336878 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 14 21:38:28.342918 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.342896 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 14 21:38:28.349462 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.349420 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:28.362554 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.362521 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r"] Jul 14 21:38:28.365415 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.365388 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pcscn" event={"ID":"6c40e5ea-9442-4e32-8636-999f09c6cee6","Type":"ContainerStarted","Data":"9ea36720989063a73b419e0a076ae0284318443eb9dc69392d371961dde3b9e1"} Jul 14 21:38:28.418853 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.418810 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-xrqvn"] Jul 14 21:38:28.420045 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.420011 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t69q8\" (UniqueName: \"kubernetes.io/projected/9d72f25b-5dcc-4d47-b7bb-bb4df90799c9-kube-api-access-t69q8\") pod \"klusterlet-addon-workmgr-5877b778d9-pm87r\" (UID: \"9d72f25b-5dcc-4d47-b7bb-bb4df90799c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r" Jul 14 21:38:28.420240 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.420078 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d72f25b-5dcc-4d47-b7bb-bb4df90799c9-tmp\") pod \"klusterlet-addon-workmgr-5877b778d9-pm87r\" (UID: \"9d72f25b-5dcc-4d47-b7bb-bb4df90799c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r" Jul 14 21:38:28.420240 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.420159 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9d72f25b-5dcc-4d47-b7bb-bb4df90799c9-klusterlet-config\") pod \"klusterlet-addon-workmgr-5877b778d9-pm87r\" (UID: \"9d72f25b-5dcc-4d47-b7bb-bb4df90799c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r" Jul 14 21:38:28.422504 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.422481 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-xrqvn" Jul 14 21:38:28.427014 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.426989 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 21:38:28.427132 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.427060 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 21:38:28.427228 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.427152 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-5748l\"" Jul 14 21:38:28.454966 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.454936 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-xrqvn"] Jul 14 21:38:28.476247 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.476209 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:28.520745 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.520721 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9d72f25b-5dcc-4d47-b7bb-bb4df90799c9-klusterlet-config\") pod \"klusterlet-addon-workmgr-5877b778d9-pm87r\" (UID: \"9d72f25b-5dcc-4d47-b7bb-bb4df90799c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r" Jul 14 21:38:28.520854 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.520805 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm2mb\" (UniqueName: \"kubernetes.io/projected/087cfd76-4d93-43b7-b145-9915a91d00fd-kube-api-access-vm2mb\") pod \"downloads-68c9cbb497-xrqvn\" (UID: \"087cfd76-4d93-43b7-b145-9915a91d00fd\") " pod="openshift-console/downloads-68c9cbb497-xrqvn" Jul 14 21:38:28.520910 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.520876 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t69q8\" (UniqueName: \"kubernetes.io/projected/9d72f25b-5dcc-4d47-b7bb-bb4df90799c9-kube-api-access-t69q8\") pod \"klusterlet-addon-workmgr-5877b778d9-pm87r\" (UID: \"9d72f25b-5dcc-4d47-b7bb-bb4df90799c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r" Jul 14 21:38:28.520973 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.520924 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d72f25b-5dcc-4d47-b7bb-bb4df90799c9-tmp\") pod \"klusterlet-addon-workmgr-5877b778d9-pm87r\" (UID: \"9d72f25b-5dcc-4d47-b7bb-bb4df90799c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r" Jul 14 21:38:28.521312 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.521292 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d72f25b-5dcc-4d47-b7bb-bb4df90799c9-tmp\") pod \"klusterlet-addon-workmgr-5877b778d9-pm87r\" (UID: \"9d72f25b-5dcc-4d47-b7bb-bb4df90799c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r" Jul 14 21:38:28.523312 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.523292 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9d72f25b-5dcc-4d47-b7bb-bb4df90799c9-klusterlet-config\") pod \"klusterlet-addon-workmgr-5877b778d9-pm87r\" (UID: \"9d72f25b-5dcc-4d47-b7bb-bb4df90799c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r" Jul 14 21:38:28.547390 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.547364 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t69q8\" (UniqueName: \"kubernetes.io/projected/9d72f25b-5dcc-4d47-b7bb-bb4df90799c9-kube-api-access-t69q8\") pod \"klusterlet-addon-workmgr-5877b778d9-pm87r\" (UID: \"9d72f25b-5dcc-4d47-b7bb-bb4df90799c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r" Jul 14 21:38:28.563526 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.563494 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:28.621909 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.621879 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm2mb\" (UniqueName: \"kubernetes.io/projected/087cfd76-4d93-43b7-b145-9915a91d00fd-kube-api-access-vm2mb\") pod \"downloads-68c9cbb497-xrqvn\" (UID: \"087cfd76-4d93-43b7-b145-9915a91d00fd\") " pod="openshift-console/downloads-68c9cbb497-xrqvn" Jul 14 21:38:28.622077 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.621921 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-8xwjm\" (UID: \"be6e2b61-604a-43e4-a2e4-4379dbaf0b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" Jul 14 21:38:28.624267 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.624246 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be6e2b61-604a-43e4-a2e4-4379dbaf0b9c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-8xwjm\" (UID: \"be6e2b61-604a-43e4-a2e4-4379dbaf0b9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" Jul 14 21:38:28.637871 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.637846 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm2mb\" (UniqueName: \"kubernetes.io/projected/087cfd76-4d93-43b7-b145-9915a91d00fd-kube-api-access-vm2mb\") pod \"downloads-68c9cbb497-xrqvn\" (UID: \"087cfd76-4d93-43b7-b145-9915a91d00fd\") " pod="openshift-console/downloads-68c9cbb497-xrqvn" Jul 14 21:38:28.640660 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.640644 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r" Jul 14 21:38:28.663937 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.663897 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:28.722805 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.722771 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-82wc4\" (UID: \"587d0871-09d8-4223-955c-52d205e0051b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" Jul 14 21:38:28.722934 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.722825 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:38:28.722934 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.722894 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:38:28.723580 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.723560 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-service-ca-bundle\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:38:28.726324 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.726299 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ed9b70-c79d-44a2-950c-af1d0b6bdd57-metrics-certs\") pod \"router-default-5c67f54db8-mlqfz\" (UID: \"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57\") " pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:38:28.726324 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.726325 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/587d0871-09d8-4223-955c-52d205e0051b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-82wc4\" (UID: \"587d0871-09d8-4223-955c-52d205e0051b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" Jul 14 21:38:28.731155 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.731133 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-xrqvn" Jul 14 21:38:28.764225 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.764167 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:28.769070 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.769042 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r"] Jul 14 21:38:28.772098 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:38:28.772074 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d72f25b_5dcc_4d47_b7bb_bb4df90799c9.slice/crio-81a6409222a191c97e9ccd4440b3c4cf9ebe410a2247c616b41af786fe548105 WatchSource:0}: Error finding container 81a6409222a191c97e9ccd4440b3c4cf9ebe410a2247c616b41af786fe548105: Status 404 returned error can't find the container with id 81a6409222a191c97e9ccd4440b3c4cf9ebe410a2247c616b41af786fe548105 Jul 14 21:38:28.819570 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.819217 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fn6sz\"" Jul 14 21:38:28.824237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.823445 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" Jul 14 21:38:28.824237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.823752 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8cggq\" (UID: \"c03004c3-29b8-4813-85e1-5df04bd12a34\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8cggq" Jul 14 21:38:28.826704 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.826667 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c03004c3-29b8-4813-85e1-5df04bd12a34-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8cggq\" (UID: \"c03004c3-29b8-4813-85e1-5df04bd12a34\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8cggq" Jul 14 21:38:28.876486 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.876147 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-xrqvn"] Jul 14 21:38:28.880591 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.878296 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:28.880591 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.878899 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-k947l\"" Jul 14 21:38:28.880591 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:38:28.879476 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087cfd76_4d93_43b7_b145_9915a91d00fd.slice/crio-52ff1710080c298a33ad945d860ffed9deb962e5fe41cf4cb5cf0ff5d378acfa WatchSource:0}: Error finding container 52ff1710080c298a33ad945d860ffed9deb962e5fe41cf4cb5cf0ff5d378acfa: Status 404 returned error can't find the container with id 52ff1710080c298a33ad945d860ffed9deb962e5fe41cf4cb5cf0ff5d378acfa Jul 14 21:38:28.887083 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.886610 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" Jul 14 21:38:28.907048 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.907017 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-sxltv\"" Jul 14 21:38:28.918828 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.918585 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:38:28.973310 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.973085 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-z2lz9\"" Jul 14 21:38:28.976153 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.976081 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm"] Jul 14 21:38:28.983062 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:28.982798 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-8cggq" Jul 14 21:38:29.093866 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:29.093751 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4"] Jul 14 21:38:29.096807 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:38:29.096772 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587d0871_09d8_4223_955c_52d205e0051b.slice/crio-6cbde7eaba634f0de2bffe6a75cdf26e75a901fe807f40870a4e5723a674eb94 WatchSource:0}: Error finding container 6cbde7eaba634f0de2bffe6a75cdf26e75a901fe807f40870a4e5723a674eb94: Status 404 returned error can't find the container with id 6cbde7eaba634f0de2bffe6a75cdf26e75a901fe807f40870a4e5723a674eb94 Jul 14 21:38:29.134240 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:29.131939 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c67f54db8-mlqfz"] Jul 14 21:38:29.136830 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:38:29.136790 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ed9b70_c79d_44a2_950c_af1d0b6bdd57.slice/crio-904fc0e603572a02f568f83deeb01db571235faae9a1d5e8b66a14ed57f50fba WatchSource:0}: Error finding container 904fc0e603572a02f568f83deeb01db571235faae9a1d5e8b66a14ed57f50fba: Status 404 returned error can't find the container with id 904fc0e603572a02f568f83deeb01db571235faae9a1d5e8b66a14ed57f50fba Jul 14 21:38:29.172106 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:29.171972 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-8cggq"] Jul 14 21:38:29.175175 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:38:29.175144 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03004c3_29b8_4813_85e1_5df04bd12a34.slice/crio-cc923d4fc0794bbc529f10cbd5a8865e8c7b15299f34002c7c8334a3d57235e4 WatchSource:0}: Error finding container cc923d4fc0794bbc529f10cbd5a8865e8c7b15299f34002c7c8334a3d57235e4: Status 404 returned error can't find the container with id cc923d4fc0794bbc529f10cbd5a8865e8c7b15299f34002c7c8334a3d57235e4 Jul 14 21:38:29.372799 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:29.372757 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pcscn" event={"ID":"6c40e5ea-9442-4e32-8636-999f09c6cee6","Type":"ContainerStarted","Data":"e0cb92760da2746d149b6c1f5e0b1d062e0f1333ed67ccd838d2fbdd4902cf05"} Jul 14 21:38:29.374256 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:29.374221 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" event={"ID":"587d0871-09d8-4223-955c-52d205e0051b","Type":"ContainerStarted","Data":"6cbde7eaba634f0de2bffe6a75cdf26e75a901fe807f40870a4e5723a674eb94"} Jul 14 21:38:29.375640 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:29.375589 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-xrqvn" event={"ID":"087cfd76-4d93-43b7-b145-9915a91d00fd","Type":"ContainerStarted","Data":"52ff1710080c298a33ad945d860ffed9deb962e5fe41cf4cb5cf0ff5d378acfa"} Jul 14 21:38:29.377683 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:29.377653 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c67f54db8-mlqfz" event={"ID":"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57","Type":"ContainerStarted","Data":"94fac5eccee90af05c210aa93501fa46a5b218d57751b3207d79c0f4bbc792cf"} Jul 14 21:38:29.377805 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:29.377706 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c67f54db8-mlqfz" event={"ID":"b1ed9b70-c79d-44a2-950c-af1d0b6bdd57","Type":"ContainerStarted","Data":"904fc0e603572a02f568f83deeb01db571235faae9a1d5e8b66a14ed57f50fba"} Jul 14 21:38:29.379341 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:29.379311 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" event={"ID":"be6e2b61-604a-43e4-a2e4-4379dbaf0b9c","Type":"ContainerStarted","Data":"0605bac76aa6e21a3eed0663ce5764291985875c64c7ed75f436d99532a9b9f3"} Jul 14 21:38:29.381889 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:29.381859 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-8cggq" event={"ID":"c03004c3-29b8-4813-85e1-5df04bd12a34","Type":"ContainerStarted","Data":"cc923d4fc0794bbc529f10cbd5a8865e8c7b15299f34002c7c8334a3d57235e4"} Jul 14 21:38:29.383266 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:29.383142 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r" event={"ID":"9d72f25b-5dcc-4d47-b7bb-bb4df90799c9","Type":"ContainerStarted","Data":"81a6409222a191c97e9ccd4440b3c4cf9ebe410a2247c616b41af786fe548105"} Jul 14 21:38:29.404493 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:29.404381 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5c67f54db8-mlqfz" podStartSLOduration=33.404363329 podStartE2EDuration="33.404363329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:29.402872603 +0000 UTC m=+75.941578315" watchObservedRunningTime="2026-07-14 21:38:29.404363329 +0000 UTC m=+75.943069001" Jul 14 21:38:29.919836 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:29.919434 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:38:29.922648 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:29.922620 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:38:30.393745 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:30.393651 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:38:30.393931 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:30.393827 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5c67f54db8-mlqfz" Jul 14 21:38:33.994131 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:33.994094 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-754dc5fd4f-b4ksg"] Jul 14 21:38:34.022338 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.022303 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-754dc5fd4f-b4ksg"] Jul 14 21:38:34.022517 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.022408 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.026045 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.026011 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 21:38:34.026574 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.026297 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 21:38:34.026715 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.026655 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 21:38:34.026826 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.026803 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 21:38:34.027105 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.027083 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6thlj\"" Jul 14 21:38:34.027217 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.027096 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 21:38:34.174259 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.174220 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-serving-cert\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.174259 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.174282 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-oauth-config\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.174513 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.174389 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-service-ca\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.174560 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.174510 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-oauth-serving-cert\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.174560 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.174545 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl6mn\" (UniqueName: \"kubernetes.io/projected/ddedac96-3b61-4954-868f-19cb76d6dc4c-kube-api-access-wl6mn\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.174667 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.174573 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-config\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.275172 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.275087 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-service-ca\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.275172 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.275143 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-oauth-serving-cert\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.275426 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.275222 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl6mn\" (UniqueName: \"kubernetes.io/projected/ddedac96-3b61-4954-868f-19cb76d6dc4c-kube-api-access-wl6mn\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.275426 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.275252 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-config\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.275426 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.275293 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-serving-cert\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.275426 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.275330 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-oauth-config\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.275993 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.275963 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-service-ca\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.276120 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.275963 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-oauth-serving-cert\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.276335 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.276276 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-config\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.278686 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.278652 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-oauth-config\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.278792 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.278656 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-serving-cert\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.285979 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.285954 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl6mn\" (UniqueName: \"kubernetes.io/projected/ddedac96-3b61-4954-868f-19cb76d6dc4c-kube-api-access-wl6mn\") pod \"console-754dc5fd4f-b4ksg\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.333927 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.333890 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:34.371648 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.371513 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:34.569305 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.569211 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:34.630398 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.630356 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:34.802102 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.802063 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:34.860345 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:34.860317 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:35.361153 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:35.361121 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jrvlv" Jul 14 21:38:36.385435 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:36.385297 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:36.488406 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:36.488164 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:36.588702 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:36.588675 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-754dc5fd4f-b4ksg"] Jul 14 21:38:36.811014 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:38:36.810924 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddedac96_3b61_4954_868f_19cb76d6dc4c.slice/crio-36e803e791069c0aa367bd194e8dd4d8413cb1a2bbd7204cd6e4db681a02a438 WatchSource:0}: Error finding container 36e803e791069c0aa367bd194e8dd4d8413cb1a2bbd7204cd6e4db681a02a438: Status 404 returned error can't find the container with id 36e803e791069c0aa367bd194e8dd4d8413cb1a2bbd7204cd6e4db681a02a438 Jul 14 21:38:36.906807 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:36.906772 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:37.007548 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.007449 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:37.100568 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.100529 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:37.225778 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.225394 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:37.437014 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.436913 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pcscn" event={"ID":"6c40e5ea-9442-4e32-8636-999f09c6cee6","Type":"ContainerStarted","Data":"a7ef12397837d6c5f8fc9c40f10b5370765daaff0f444bcc8bc26d24f1fa749f"} Jul 14 21:38:37.440105 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.439679 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" event={"ID":"587d0871-09d8-4223-955c-52d205e0051b","Type":"ContainerStarted","Data":"c469da5a555fee40bc07da3e3d052655e740e42eeed85e360b70bcc24af36729"} Jul 14 21:38:37.441482 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.441447 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-754dc5fd4f-b4ksg" event={"ID":"ddedac96-3b61-4954-868f-19cb76d6dc4c","Type":"ContainerStarted","Data":"36e803e791069c0aa367bd194e8dd4d8413cb1a2bbd7204cd6e4db681a02a438"} Jul 14 21:38:37.445182 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.445155 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" event={"ID":"be6e2b61-604a-43e4-a2e4-4379dbaf0b9c","Type":"ContainerStarted","Data":"9fa81050ec0b036ee2dfb1bfdce4d625c94661fa0ef4521613666fdabd521fa1"} Jul 14 21:38:37.445842 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.445222 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" event={"ID":"be6e2b61-604a-43e4-a2e4-4379dbaf0b9c","Type":"ContainerStarted","Data":"4ab311e69ee621d193cb7f6b3009025efb598c6628ab622b4c3635322133783e"} Jul 14 21:38:37.447453 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.447425 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-8cggq" event={"ID":"c03004c3-29b8-4813-85e1-5df04bd12a34","Type":"ContainerStarted","Data":"4103c19ea6eca6b37e87a56b5620a64063d6948248d25570d41c85bb404f704d"} Jul 14 21:38:37.449860 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.449831 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r" event={"ID":"9d72f25b-5dcc-4d47-b7bb-bb4df90799c9","Type":"ContainerStarted","Data":"1848849487b8d3642247ad64850290fbfc52d36d3f82ebbf8a168538624b0255"} Jul 14 21:38:37.450633 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.450612 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r" Jul 14 21:38:37.452541 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.452524 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r" Jul 14 21:38:37.458154 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.457016 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pcscn" podStartSLOduration=18.256797573 podStartE2EDuration="32.456997137s" totalImagesPullingTime="14.200199564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 21:38:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:37.455165189 +0000 UTC m=+83.993870863" watchObservedRunningTime="2026-07-14 21:38:37.456997137 +0000 UTC m=+83.995702811" Jul 14 21:38:37.471808 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.471736 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-8cggq" podStartSLOduration=34.220207301 podStartE2EDuration="41.471717942s" totalImagesPullingTime="7.251510641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:37.471283251 +0000 UTC m=+84.009988925" watchObservedRunningTime="2026-07-14 21:38:37.471717942 +0000 UTC m=+84.010423613" Jul 14 21:38:37.492085 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.492018 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-82wc4" podStartSLOduration=34.158100789 podStartE2EDuration="41.491995077s" totalImagesPullingTime="7.333894288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:37.490613205 +0000 UTC m=+84.029318880" watchObservedRunningTime="2026-07-14 21:38:37.491995077 +0000 UTC m=+84.030700752" Jul 14 21:38:37.514811 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.513934 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5877b778d9-pm87r" podStartSLOduration=1.429843054 podStartE2EDuration="9.513909807s" totalImagesPullingTime="8.084066753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:37.510635869 +0000 UTC m=+84.049341543" watchObservedRunningTime="2026-07-14 21:38:37.513909807 +0000 UTC m=+84.052615481" Jul 14 21:38:37.529227 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.528990 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8xwjm" podStartSLOduration=34.140210466 podStartE2EDuration="41.528967585s" totalImagesPullingTime="7.388757119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:37.527779487 +0000 UTC m=+84.066485159" watchObservedRunningTime="2026-07-14 21:38:37.528967585 +0000 UTC m=+84.067673261" Jul 14 21:38:37.608708 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.608592 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:37.675117 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:37.675079 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:38.081423 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:38.081329 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:38.177990 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:38.177953 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:38.378156 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:38.378126 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:38.438840 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:38.438790 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:39.286871 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:39.286816 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:39.362233 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:39.362121 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:39.947967 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:39.947927 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:40.047229 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:40.047185 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:40.147330 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:40.147291 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:40.464728 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:40.464689 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-754dc5fd4f-b4ksg" event={"ID":"ddedac96-3b61-4954-868f-19cb76d6dc4c","Type":"ContainerStarted","Data":"046d0e53c7f77e33e83ecac3ab0ab9c33dfebf5d1eeb557b309f7363487f57f8"} Jul 14 21:38:40.494632 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:40.493657 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-754dc5fd4f-b4ksg" podStartSLOduration=4.355144706 podStartE2EDuration="7.493634848s" totalImagesPullingTime="3.138490142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:40.490123523 +0000 UTC m=+87.028829210" watchObservedRunningTime="2026-07-14 21:38:40.493634848 +0000 UTC m=+87.032340521" Jul 14 21:38:41.593401 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:41.593323 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:41.666165 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:41.666117 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:42.276255 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:42.276104 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:42.338586 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:42.338539 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:38:42.372401 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:42.372357 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:42.440007 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:42.439971 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:42.646851 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:42.646810 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:42.710226 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:42.710171 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:43.121273 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:43.121231 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:43.197001 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:43.196966 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:43.574411 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:43.574322 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:44.334635 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:44.334598 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:44.335260 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:44.334676 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:44.340508 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:44.340481 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:44.488435 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:44.488356 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:38:46.494731 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:46.494691 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:47.242124 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:47.242088 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:47.340743 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:47.340708 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:47.410458 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:47.410413 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:48.290863 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:48.290820 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:48.392052 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:48.392016 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:48.502259 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:48.502217 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-xrqvn" event={"ID":"087cfd76-4d93-43b7-b145-9915a91d00fd","Type":"ContainerStarted","Data":"4ae39628e33e9994023454f076150e88cbf56bbf608de05dfa3d40f52a40b8fc"} Jul 14 21:38:48.502464 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:48.502441 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-xrqvn" Jul 14 21:38:48.519619 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:48.519585 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-xrqvn" Jul 14 21:38:48.525999 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:48.525921 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-xrqvn" podStartSLOduration=1.644812431 podStartE2EDuration="20.525904695s" totalImagesPullingTime="18.881092264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:48.524499741 +0000 UTC m=+95.063205426" watchObservedRunningTime="2026-07-14 21:38:48.525904695 +0000 UTC m=+95.064610369" Jul 14 21:38:49.066814 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:49.066763 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:49.133364 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:49.133331 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:49.314120 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:49.314079 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:49.592307 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:49.592268 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:49.797664 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:49.797625 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jlt6n"] Jul 14 21:38:49.806978 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:49.806935 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:49.825720 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:49.825690 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:49.837870 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:49.837842 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 21:38:49.838021 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:49.837853 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 21:38:49.838021 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:49.837905 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 21:38:49.838741 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:49.838717 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b4qtd\"" Jul 14 21:38:49.838854 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:49.838754 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 21:38:49.958069 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:49.957972 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:50.015362 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.015326 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/175e5d82-2c32-4067-90be-cccc501c0990-sys\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.015550 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.015377 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/175e5d82-2c32-4067-90be-cccc501c0990-node-exporter-textfile\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.015550 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.015407 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/175e5d82-2c32-4067-90be-cccc501c0990-node-exporter-wtmp\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.015550 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.015457 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/175e5d82-2c32-4067-90be-cccc501c0990-node-exporter-tls\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.015550 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.015480 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89pvj\" (UniqueName: \"kubernetes.io/projected/175e5d82-2c32-4067-90be-cccc501c0990-kube-api-access-89pvj\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.015550 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.015513 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/175e5d82-2c32-4067-90be-cccc501c0990-metrics-client-ca\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.015550 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.015541 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/175e5d82-2c32-4067-90be-cccc501c0990-node-exporter-accelerators-collector-config\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.015851 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.015640 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/175e5d82-2c32-4067-90be-cccc501c0990-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.015851 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.015689 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/175e5d82-2c32-4067-90be-cccc501c0990-root\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.108939 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.108899 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:50.116500 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.116465 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/175e5d82-2c32-4067-90be-cccc501c0990-sys\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.116670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.116517 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/175e5d82-2c32-4067-90be-cccc501c0990-node-exporter-textfile\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.116670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.116541 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/175e5d82-2c32-4067-90be-cccc501c0990-node-exporter-wtmp\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.116670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.116576 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/175e5d82-2c32-4067-90be-cccc501c0990-sys\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.116670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.116586 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/175e5d82-2c32-4067-90be-cccc501c0990-node-exporter-tls\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.116670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.116626 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89pvj\" (UniqueName: \"kubernetes.io/projected/175e5d82-2c32-4067-90be-cccc501c0990-kube-api-access-89pvj\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.116670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.116666 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/175e5d82-2c32-4067-90be-cccc501c0990-metrics-client-ca\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.117045 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.116697 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/175e5d82-2c32-4067-90be-cccc501c0990-node-exporter-accelerators-collector-config\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.117045 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.116738 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/175e5d82-2c32-4067-90be-cccc501c0990-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.117045 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.116776 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/175e5d82-2c32-4067-90be-cccc501c0990-node-exporter-wtmp\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.117045 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.116787 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/175e5d82-2c32-4067-90be-cccc501c0990-root\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.117045 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.116871 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/175e5d82-2c32-4067-90be-cccc501c0990-root\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.117045 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.116942 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/175e5d82-2c32-4067-90be-cccc501c0990-node-exporter-textfile\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.117513 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.117475 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/175e5d82-2c32-4067-90be-cccc501c0990-node-exporter-accelerators-collector-config\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.118011 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.117985 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/175e5d82-2c32-4067-90be-cccc501c0990-metrics-client-ca\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.119831 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.119759 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/175e5d82-2c32-4067-90be-cccc501c0990-node-exporter-tls\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.120730 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.120704 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/175e5d82-2c32-4067-90be-cccc501c0990-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.131616 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.131588 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89pvj\" (UniqueName: \"kubernetes.io/projected/175e5d82-2c32-4067-90be-cccc501c0990-kube-api-access-89pvj\") pod \"node-exporter-jlt6n\" (UID: \"175e5d82-2c32-4067-90be-cccc501c0990\") " pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.137455 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.137423 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jlt6n" Jul 14 21:38:50.177577 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:38:50.177541 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175e5d82_2c32_4067_90be_cccc501c0990.slice/crio-dfeb16c69c7a974c1cae98016f74aa7b8eaa069227d16c54acf17c549b2fcaf0 WatchSource:0}: Error finding container dfeb16c69c7a974c1cae98016f74aa7b8eaa069227d16c54acf17c549b2fcaf0: Status 404 returned error can't find the container with id dfeb16c69c7a974c1cae98016f74aa7b8eaa069227d16c54acf17c549b2fcaf0 Jul 14 21:38:50.215596 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.215517 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:50.457836 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.457797 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:50.513940 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.513846 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jlt6n" event={"ID":"175e5d82-2c32-4067-90be-cccc501c0990","Type":"ContainerStarted","Data":"dfeb16c69c7a974c1cae98016f74aa7b8eaa069227d16c54acf17c549b2fcaf0"} Jul 14 21:38:50.561615 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.561581 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:50.659806 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.659760 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:50.746074 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:50.745996 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:51.215327 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:51.215286 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cf444f6dc-f2bc5"] Jul 14 21:38:51.300147 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:51.300074 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:51.322871 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:51.322842 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8hxtk" Jul 14 21:38:51.514882 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:51.514777 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:51.615407 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:51.615367 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:52.527401 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:52.527361 2552 generic.go:356] "Generic (PLEG): container finished" podID="175e5d82-2c32-4067-90be-cccc501c0990" containerID="f9b91df0da3eed7c0b6ab80fda1a652c62554b970cd2b7b26e62bd84e471ed6a" exitCode=0 Jul 14 21:38:52.528011 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:52.527424 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jlt6n" event={"ID":"175e5d82-2c32-4067-90be-cccc501c0990","Type":"ContainerDied","Data":"f9b91df0da3eed7c0b6ab80fda1a652c62554b970cd2b7b26e62bd84e471ed6a"} Jul 14 21:38:52.570450 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:52.570410 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:52.633890 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:52.633857 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:52.713859 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:52.713836 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:52.814665 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:52.814631 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:53.535546 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:53.535516 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jlt6n" event={"ID":"175e5d82-2c32-4067-90be-cccc501c0990","Type":"ContainerStarted","Data":"a5bebcd6867504c1806fda06ccfdec8ca24a21b1df4d4c7a64ee3495a8251f14"} Jul 14 21:38:53.536101 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:53.535575 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jlt6n" event={"ID":"175e5d82-2c32-4067-90be-cccc501c0990","Type":"ContainerStarted","Data":"3efdb2af2849287e7bdddf0a859992fcbdb3449c05b1fb0fee789d744cd8587f"} Jul 14 21:38:53.560391 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:53.560316 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jlt6n" podStartSLOduration=2.8786307669999998 podStartE2EDuration="4.560293879s" totalImagesPullingTime="1.565271306s" totalInitContainerRuntime="116.391806ms" isStatefulPod=true podCreationTimestamp="2026-07-14 21:38:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:38:53.558560497 +0000 UTC m=+100.097266170" watchObservedRunningTime="2026-07-14 21:38:53.560293879 +0000 UTC m=+100.098999551" Jul 14 21:38:53.609635 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:53.609592 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:53.710977 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:53.710929 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:54.366176 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:54.366139 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:54.432255 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:54.432214 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:54.521953 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:54.521913 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:55.921507 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:55.921472 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:56.026308 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:56.026272 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:38:56.092042 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:38:56.091997 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:39:00.201998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:00.201941 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:39:00.264539 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:00.264503 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:39:00.442679 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:00.442639 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:39:00.507535 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:00.507451 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:39:04.797804 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:04.797757 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:39:04.898327 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:04.898281 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:39:06.511856 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:06.511820 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-754dc5fd4f-b4ksg"] Jul 14 21:39:06.621073 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:06.621018 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:39:06.730386 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:06.730353 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:39:06.831682 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:06.831594 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:39:16.245993 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.245947 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" podUID="547803e8-dd34-48fb-bb45-f08182623df4" containerName="registry" containerID="cri-o://b2abe284fb1c80b3c0199a94472b109e74f4cd707638cca424d83168b356b357" gracePeriod=30 Jul 14 21:39:16.520168 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.520095 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:39:16.627521 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.627490 2552 generic.go:356] "Generic (PLEG): container finished" podID="547803e8-dd34-48fb-bb45-f08182623df4" containerID="b2abe284fb1c80b3c0199a94472b109e74f4cd707638cca424d83168b356b357" exitCode=0 Jul 14 21:39:16.627689 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.627533 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" Jul 14 21:39:16.627689 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.627566 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" event={"ID":"547803e8-dd34-48fb-bb45-f08182623df4","Type":"ContainerDied","Data":"b2abe284fb1c80b3c0199a94472b109e74f4cd707638cca424d83168b356b357"} Jul 14 21:39:16.627689 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.627606 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cf444f6dc-f2bc5" event={"ID":"547803e8-dd34-48fb-bb45-f08182623df4","Type":"ContainerDied","Data":"4f83ccf14e00d2d14bdb3068dc6aeff2db0e8566600d8fb43b240c7780bc0d3d"} Jul 14 21:39:16.627689 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.627627 2552 scope.go:117] "RemoveContainer" containerID="b2abe284fb1c80b3c0199a94472b109e74f4cd707638cca424d83168b356b357" Jul 14 21:39:16.635730 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.635705 2552 scope.go:117] "RemoveContainer" containerID="b2abe284fb1c80b3c0199a94472b109e74f4cd707638cca424d83168b356b357" Jul 14 21:39:16.635992 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:39:16.635972 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2abe284fb1c80b3c0199a94472b109e74f4cd707638cca424d83168b356b357\": container with ID starting with b2abe284fb1c80b3c0199a94472b109e74f4cd707638cca424d83168b356b357 not found: ID does not exist" containerID="b2abe284fb1c80b3c0199a94472b109e74f4cd707638cca424d83168b356b357" Jul 14 21:39:16.636042 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.636006 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2abe284fb1c80b3c0199a94472b109e74f4cd707638cca424d83168b356b357"} err="failed to get container status \"b2abe284fb1c80b3c0199a94472b109e74f4cd707638cca424d83168b356b357\": rpc error: code = NotFound desc = could not find container \"b2abe284fb1c80b3c0199a94472b109e74f4cd707638cca424d83168b356b357\": container with ID starting with b2abe284fb1c80b3c0199a94472b109e74f4cd707638cca424d83168b356b357 not found: ID does not exist" Jul 14 21:39:16.647361 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.647334 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-bound-sa-token\") pod \"547803e8-dd34-48fb-bb45-f08182623df4\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " Jul 14 21:39:16.647462 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.647377 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/547803e8-dd34-48fb-bb45-f08182623df4-ca-trust-extracted\") pod \"547803e8-dd34-48fb-bb45-f08182623df4\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " Jul 14 21:39:16.647462 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.647414 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/547803e8-dd34-48fb-bb45-f08182623df4-trusted-ca\") pod \"547803e8-dd34-48fb-bb45-f08182623df4\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " Jul 14 21:39:16.647462 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.647434 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/547803e8-dd34-48fb-bb45-f08182623df4-installation-pull-secrets\") pod \"547803e8-dd34-48fb-bb45-f08182623df4\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " Jul 14 21:39:16.647462 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.647454 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/547803e8-dd34-48fb-bb45-f08182623df4-image-registry-private-configuration\") pod \"547803e8-dd34-48fb-bb45-f08182623df4\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " Jul 14 21:39:16.647621 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.647476 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/547803e8-dd34-48fb-bb45-f08182623df4-registry-certificates\") pod \"547803e8-dd34-48fb-bb45-f08182623df4\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " Jul 14 21:39:16.647621 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.647525 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls\") pod \"547803e8-dd34-48fb-bb45-f08182623df4\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " Jul 14 21:39:16.647621 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.647554 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjhtb\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-kube-api-access-bjhtb\") pod \"547803e8-dd34-48fb-bb45-f08182623df4\" (UID: \"547803e8-dd34-48fb-bb45-f08182623df4\") " Jul 14 21:39:16.648139 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.648086 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547803e8-dd34-48fb-bb45-f08182623df4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "547803e8-dd34-48fb-bb45-f08182623df4" (UID: "547803e8-dd34-48fb-bb45-f08182623df4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:39:16.648280 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.648184 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547803e8-dd34-48fb-bb45-f08182623df4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "547803e8-dd34-48fb-bb45-f08182623df4" (UID: "547803e8-dd34-48fb-bb45-f08182623df4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:39:16.650069 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.650030 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "547803e8-dd34-48fb-bb45-f08182623df4" (UID: "547803e8-dd34-48fb-bb45-f08182623df4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:39:16.650172 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.650084 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547803e8-dd34-48fb-bb45-f08182623df4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "547803e8-dd34-48fb-bb45-f08182623df4" (UID: "547803e8-dd34-48fb-bb45-f08182623df4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:39:16.650172 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.650131 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "547803e8-dd34-48fb-bb45-f08182623df4" (UID: "547803e8-dd34-48fb-bb45-f08182623df4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:39:16.650283 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.650221 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547803e8-dd34-48fb-bb45-f08182623df4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "547803e8-dd34-48fb-bb45-f08182623df4" (UID: "547803e8-dd34-48fb-bb45-f08182623df4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:39:16.650381 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.650366 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-kube-api-access-bjhtb" (OuterVolumeSpecName: "kube-api-access-bjhtb") pod "547803e8-dd34-48fb-bb45-f08182623df4" (UID: "547803e8-dd34-48fb-bb45-f08182623df4"). InnerVolumeSpecName "kube-api-access-bjhtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:39:16.656043 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.656017 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547803e8-dd34-48fb-bb45-f08182623df4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "547803e8-dd34-48fb-bb45-f08182623df4" (UID: "547803e8-dd34-48fb-bb45-f08182623df4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:39:16.748573 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.748540 2552 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-bound-sa-token\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:39:16.748573 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.748569 2552 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/547803e8-dd34-48fb-bb45-f08182623df4-ca-trust-extracted\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:39:16.748573 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.748578 2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/547803e8-dd34-48fb-bb45-f08182623df4-trusted-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:39:16.748814 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.748588 2552 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/547803e8-dd34-48fb-bb45-f08182623df4-installation-pull-secrets\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:39:16.748814 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.748598 2552 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/547803e8-dd34-48fb-bb45-f08182623df4-image-registry-private-configuration\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:39:16.748814 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.748607 2552 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/547803e8-dd34-48fb-bb45-f08182623df4-registry-certificates\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:39:16.748814 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.748617 2552 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-registry-tls\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:39:16.748814 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.748625 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjhtb\" (UniqueName: \"kubernetes.io/projected/547803e8-dd34-48fb-bb45-f08182623df4-kube-api-access-bjhtb\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:39:16.951471 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.951435 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cf444f6dc-f2bc5"] Jul 14 21:39:16.956202 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:16.956162 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cf444f6dc-f2bc5"] Jul 14 21:39:17.990502 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:17.990467 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="547803e8-dd34-48fb-bb45-f08182623df4" path="/var/lib/kubelet/pods/547803e8-dd34-48fb-bb45-f08182623df4/volumes" Jul 14 21:39:19.808147 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:19.808114 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:39:19.888013 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:19.887961 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:39:24.659144 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:24.659056 2552 generic.go:356] "Generic (PLEG): container finished" podID="c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3" containerID="cf92dc9d3dc033fb0691269fb1ba03a0df9c65e9ac54990d966df6fc102d61d2" exitCode=0 Jul 14 21:39:24.659559 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:24.659133 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64" event={"ID":"c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3","Type":"ContainerDied","Data":"cf92dc9d3dc033fb0691269fb1ba03a0df9c65e9ac54990d966df6fc102d61d2"} Jul 14 21:39:24.659559 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:24.659489 2552 scope.go:117] "RemoveContainer" containerID="cf92dc9d3dc033fb0691269fb1ba03a0df9c65e9ac54990d966df6fc102d61d2" Jul 14 21:39:25.664449 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:25.664414 2552 generic.go:356] "Generic (PLEG): container finished" podID="eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87" containerID="327d484008c2f5f91962532d635820f1e8c608c9d4fa215f40ba0e98db11ba69" exitCode=0 Jul 14 21:39:25.664881 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:25.664489 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" event={"ID":"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87","Type":"ContainerDied","Data":"327d484008c2f5f91962532d635820f1e8c608c9d4fa215f40ba0e98db11ba69"} Jul 14 21:39:25.664934 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:25.664880 2552 scope.go:117] "RemoveContainer" containerID="327d484008c2f5f91962532d635820f1e8c608c9d4fa215f40ba0e98db11ba69" Jul 14 21:39:25.666500 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:25.666468 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-77p64" event={"ID":"c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3","Type":"ContainerStarted","Data":"5affba72c010c4ffaa6626e1db5dd06ea66c6a5233f20cdb8b316d0bd444f433"} Jul 14 21:39:26.671856 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:26.671819 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-sgt5s" event={"ID":"eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87","Type":"ContainerStarted","Data":"f41881550417cde996ef3fde85f4b5ad8b83bab98927e29044d2c0b801823d7d"} Jul 14 21:39:29.684111 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:29.684078 2552 generic.go:356] "Generic (PLEG): container finished" podID="56c0e606-2ba2-4a4e-9438-ea86a06152a9" containerID="c1ac29fe81da688eb12a154867a41980f6012c15afa0b1bb914b9a7817e578db" exitCode=0 Jul 14 21:39:29.684508 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:29.684155 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk" event={"ID":"56c0e606-2ba2-4a4e-9438-ea86a06152a9","Type":"ContainerDied","Data":"c1ac29fe81da688eb12a154867a41980f6012c15afa0b1bb914b9a7817e578db"} Jul 14 21:39:29.684508 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:29.684504 2552 scope.go:117] "RemoveContainer" containerID="c1ac29fe81da688eb12a154867a41980f6012c15afa0b1bb914b9a7817e578db" Jul 14 21:39:30.689796 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:30.689757 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-mm7vk" event={"ID":"56c0e606-2ba2-4a4e-9438-ea86a06152a9","Type":"ContainerStarted","Data":"55ca310f726643e38ca743ee89855afa7df14911f62039663dcf6e8fe0e4f6ba"} Jul 14 21:39:31.532868 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:31.532793 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-754dc5fd4f-b4ksg" podUID="ddedac96-3b61-4954-868f-19cb76d6dc4c" containerName="console" containerID="cri-o://046d0e53c7f77e33e83ecac3ab0ab9c33dfebf5d1eeb557b309f7363487f57f8" gracePeriod=15 Jul 14 21:39:31.696426 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:31.696385 2552 generic.go:356] "Generic (PLEG): container finished" podID="ddedac96-3b61-4954-868f-19cb76d6dc4c" containerID="046d0e53c7f77e33e83ecac3ab0ab9c33dfebf5d1eeb557b309f7363487f57f8" exitCode=0 Jul 14 21:39:31.696426 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:31.696415 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-754dc5fd4f-b4ksg" event={"ID":"ddedac96-3b61-4954-868f-19cb76d6dc4c","Type":"ContainerDied","Data":"046d0e53c7f77e33e83ecac3ab0ab9c33dfebf5d1eeb557b309f7363487f57f8"} Jul 14 21:39:31.816438 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:31.816408 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:39:31.978709 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:31.978676 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-service-ca\") pod \"ddedac96-3b61-4954-868f-19cb76d6dc4c\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " Jul 14 21:39:31.978709 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:31.978713 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl6mn\" (UniqueName: \"kubernetes.io/projected/ddedac96-3b61-4954-868f-19cb76d6dc4c-kube-api-access-wl6mn\") pod \"ddedac96-3b61-4954-868f-19cb76d6dc4c\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " Jul 14 21:39:31.978960 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:31.978757 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-oauth-config\") pod \"ddedac96-3b61-4954-868f-19cb76d6dc4c\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " Jul 14 21:39:31.978960 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:31.978784 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-config\") pod \"ddedac96-3b61-4954-868f-19cb76d6dc4c\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " Jul 14 21:39:31.978960 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:31.978807 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-serving-cert\") pod \"ddedac96-3b61-4954-868f-19cb76d6dc4c\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " Jul 14 21:39:31.978960 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:31.978843 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-oauth-serving-cert\") pod \"ddedac96-3b61-4954-868f-19cb76d6dc4c\" (UID: \"ddedac96-3b61-4954-868f-19cb76d6dc4c\") " Jul 14 21:39:31.979268 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:31.979238 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-service-ca" (OuterVolumeSpecName: "service-ca") pod "ddedac96-3b61-4954-868f-19cb76d6dc4c" (UID: "ddedac96-3b61-4954-868f-19cb76d6dc4c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:39:31.979379 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:31.979336 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-config" (OuterVolumeSpecName: "console-config") pod "ddedac96-3b61-4954-868f-19cb76d6dc4c" (UID: "ddedac96-3b61-4954-868f-19cb76d6dc4c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:39:31.979428 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:31.979379 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ddedac96-3b61-4954-868f-19cb76d6dc4c" (UID: "ddedac96-3b61-4954-868f-19cb76d6dc4c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:39:31.981056 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:31.981034 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ddedac96-3b61-4954-868f-19cb76d6dc4c" (UID: "ddedac96-3b61-4954-868f-19cb76d6dc4c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:39:31.981388 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:31.981364 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ddedac96-3b61-4954-868f-19cb76d6dc4c" (UID: "ddedac96-3b61-4954-868f-19cb76d6dc4c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:39:31.981475 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:31.981382 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddedac96-3b61-4954-868f-19cb76d6dc4c-kube-api-access-wl6mn" (OuterVolumeSpecName: "kube-api-access-wl6mn") pod "ddedac96-3b61-4954-868f-19cb76d6dc4c" (UID: "ddedac96-3b61-4954-868f-19cb76d6dc4c"). InnerVolumeSpecName "kube-api-access-wl6mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:39:32.079921 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:32.079841 2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-config\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:39:32.079921 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:32.079873 2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-serving-cert\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:39:32.079921 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:32.079884 2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-oauth-serving-cert\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:39:32.079921 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:32.079893 2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddedac96-3b61-4954-868f-19cb76d6dc4c-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:39:32.079921 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:32.079901 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl6mn\" (UniqueName: \"kubernetes.io/projected/ddedac96-3b61-4954-868f-19cb76d6dc4c-kube-api-access-wl6mn\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:39:32.079921 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:32.079909 2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddedac96-3b61-4954-868f-19cb76d6dc4c-console-oauth-config\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:39:32.701652 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:32.701614 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-754dc5fd4f-b4ksg" event={"ID":"ddedac96-3b61-4954-868f-19cb76d6dc4c","Type":"ContainerDied","Data":"36e803e791069c0aa367bd194e8dd4d8413cb1a2bbd7204cd6e4db681a02a438"} Jul 14 21:39:32.702083 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:32.701668 2552 scope.go:117] "RemoveContainer" containerID="046d0e53c7f77e33e83ecac3ab0ab9c33dfebf5d1eeb557b309f7363487f57f8" Jul 14 21:39:32.702083 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:32.701628 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-754dc5fd4f-b4ksg" Jul 14 21:39:32.738386 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:32.738351 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-754dc5fd4f-b4ksg"] Jul 14 21:39:32.740088 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:32.740064 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-754dc5fd4f-b4ksg"] Jul 14 21:39:33.994687 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:33.994649 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddedac96-3b61-4954-868f-19cb76d6dc4c" path="/var/lib/kubelet/pods/ddedac96-3b61-4954-868f-19cb76d6dc4c/volumes" Jul 14 21:39:36.593401 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:36.593370 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:39:43.285496 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:39:43.285460 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:40:08.095409 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:08.095313 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:40:08.901886 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:08.901849 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:40:09.424541 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:09.424497 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:40:17.608630 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.608588 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bcb48b54d-f4nrj"] Jul 14 21:40:17.609152 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.609016 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="547803e8-dd34-48fb-bb45-f08182623df4" containerName="registry" Jul 14 21:40:17.609152 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.609035 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="547803e8-dd34-48fb-bb45-f08182623df4" containerName="registry" Jul 14 21:40:17.609152 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.609058 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddedac96-3b61-4954-868f-19cb76d6dc4c" containerName="console" Jul 14 21:40:17.609152 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.609066 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddedac96-3b61-4954-868f-19cb76d6dc4c" containerName="console" Jul 14 21:40:17.609152 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.609134 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddedac96-3b61-4954-868f-19cb76d6dc4c" containerName="console" Jul 14 21:40:17.609152 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.609146 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="547803e8-dd34-48fb-bb45-f08182623df4" containerName="registry" Jul 14 21:40:17.611605 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.611582 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.614256 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.614236 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 21:40:17.615570 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.615550 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6thlj\"" Jul 14 21:40:17.615758 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.615735 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 21:40:17.615975 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.615628 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 21:40:17.616238 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.616218 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 21:40:17.616390 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.616272 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 21:40:17.620474 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.620449 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 14 21:40:17.625717 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.625691 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bcb48b54d-f4nrj"] Jul 14 21:40:17.684998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.684948 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:40:17.736142 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.736103 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac8cd84d-9b88-44dd-a041-4824961180af-console-serving-cert\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.736338 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.736157 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-service-ca\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.736338 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.736176 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-trusted-ca-bundle\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.736338 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.736269 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chq7n\" (UniqueName: \"kubernetes.io/projected/ac8cd84d-9b88-44dd-a041-4824961180af-kube-api-access-chq7n\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.736338 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.736332 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac8cd84d-9b88-44dd-a041-4824961180af-console-oauth-config\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.736491 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.736352 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-console-config\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.736491 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.736374 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-oauth-serving-cert\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.750655 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.750619 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:40:17.836916 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.836879 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac8cd84d-9b88-44dd-a041-4824961180af-console-oauth-config\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.836916 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.836921 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-console-config\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.837167 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.836938 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-oauth-serving-cert\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.837167 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.836975 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac8cd84d-9b88-44dd-a041-4824961180af-console-serving-cert\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.837167 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.837025 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-service-ca\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.837167 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.837047 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-trusted-ca-bundle\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.837167 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.837079 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chq7n\" (UniqueName: \"kubernetes.io/projected/ac8cd84d-9b88-44dd-a041-4824961180af-kube-api-access-chq7n\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.837819 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.837794 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-console-config\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.837919 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.837894 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-oauth-serving-cert\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.838049 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.838022 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-service-ca\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.838119 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.838043 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-trusted-ca-bundle\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.839528 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.839506 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac8cd84d-9b88-44dd-a041-4824961180af-console-oauth-config\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.839668 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.839652 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac8cd84d-9b88-44dd-a041-4824961180af-console-serving-cert\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.852736 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.852707 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chq7n\" (UniqueName: \"kubernetes.io/projected/ac8cd84d-9b88-44dd-a041-4824961180af-kube-api-access-chq7n\") pod \"console-5bcb48b54d-f4nrj\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:17.923387 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:17.923297 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:18.058571 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:18.058527 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bcb48b54d-f4nrj"] Jul 14 21:40:18.060669 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:40:18.060634 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8cd84d_9b88_44dd_a041_4824961180af.slice/crio-dbb4d23c195d947c4d97558572837f0e7b1a99053bf389dc00572aed6541f319 WatchSource:0}: Error finding container dbb4d23c195d947c4d97558572837f0e7b1a99053bf389dc00572aed6541f319: Status 404 returned error can't find the container with id dbb4d23c195d947c4d97558572837f0e7b1a99053bf389dc00572aed6541f319 Jul 14 21:40:18.881725 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:18.881688 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcb48b54d-f4nrj" event={"ID":"ac8cd84d-9b88-44dd-a041-4824961180af","Type":"ContainerStarted","Data":"c885cf38e7fb93f7e00f1379c3a4776b0513348bc3c927390d7d9d9fc889acf7"} Jul 14 21:40:18.882145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:18.881740 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcb48b54d-f4nrj" event={"ID":"ac8cd84d-9b88-44dd-a041-4824961180af","Type":"ContainerStarted","Data":"dbb4d23c195d947c4d97558572837f0e7b1a99053bf389dc00572aed6541f319"} Jul 14 21:40:18.913178 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:18.913115 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bcb48b54d-f4nrj" podStartSLOduration=1.913097668 podStartE2EDuration="1.913097668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:40:18.91288376 +0000 UTC m=+185.451589432" watchObservedRunningTime="2026-07-14 21:40:18.913097668 +0000 UTC m=+185.451803340" Jul 14 21:40:24.238612 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:24.238573 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:40:24.335258 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:24.335219 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:40:25.886416 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:25.886381 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bcb48b54d-f4nrj"] Jul 14 21:40:25.926793 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:25.926760 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-664d86cc8-wmwlp"] Jul 14 21:40:25.929252 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:25.929232 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:25.963760 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:25.963729 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-664d86cc8-wmwlp"] Jul 14 21:40:25.999682 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:25.999653 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6br4t\" (UniqueName: \"kubernetes.io/projected/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-kube-api-access-6br4t\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:25.999876 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:25.999712 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-config\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:25.999876 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:25.999753 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-oauth-config\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:25.999876 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:25.999776 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-oauth-serving-cert\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:25.999876 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:25.999858 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-service-ca\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.000044 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:25.999891 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-serving-cert\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.000044 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:25.999909 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-trusted-ca-bundle\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.101158 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.101118 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-serving-cert\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.101158 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.101162 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-trusted-ca-bundle\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.101451 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.101238 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6br4t\" (UniqueName: \"kubernetes.io/projected/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-kube-api-access-6br4t\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.101451 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.101300 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-config\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.101451 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.101351 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-oauth-config\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.101451 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.101377 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-oauth-serving-cert\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.101451 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.101425 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-service-ca\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.102116 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.102091 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-service-ca\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.102254 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.102114 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-config\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.102254 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.102144 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-oauth-serving-cert\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.102550 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.102527 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-trusted-ca-bundle\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.103848 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.103823 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-oauth-config\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.103959 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.103821 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-serving-cert\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.116777 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.116733 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6br4t\" (UniqueName: \"kubernetes.io/projected/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-kube-api-access-6br4t\") pod \"console-664d86cc8-wmwlp\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.239373 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.239276 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:26.375163 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.375136 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-664d86cc8-wmwlp"] Jul 14 21:40:26.377815 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:40:26.377784 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9ecf68_daeb_4bb4_8280_247e6e8b0e4f.slice/crio-c71f6052d051787ff200d66168108835f7c9eb1aac1f75e2a4360065f4c446e7 WatchSource:0}: Error finding container c71f6052d051787ff200d66168108835f7c9eb1aac1f75e2a4360065f4c446e7: Status 404 returned error can't find the container with id c71f6052d051787ff200d66168108835f7c9eb1aac1f75e2a4360065f4c446e7 Jul 14 21:40:26.914761 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.914722 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664d86cc8-wmwlp" event={"ID":"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f","Type":"ContainerStarted","Data":"b448864302d20e23d087f35ab1ae91c7ca7882756c670f13879667d08d1baba5"} Jul 14 21:40:26.915228 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.914771 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664d86cc8-wmwlp" event={"ID":"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f","Type":"ContainerStarted","Data":"c71f6052d051787ff200d66168108835f7c9eb1aac1f75e2a4360065f4c446e7"} Jul 14 21:40:26.934975 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:26.934914 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-664d86cc8-wmwlp" podStartSLOduration=1.934897545 podStartE2EDuration="1.934897545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:40:26.934761815 +0000 UTC m=+193.473467488" watchObservedRunningTime="2026-07-14 21:40:26.934897545 +0000 UTC m=+193.473603217" Jul 14 21:40:27.923939 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:27.923910 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:36.184892 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:36.184849 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:40:36.240174 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:36.240137 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:36.240362 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:36.240230 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:36.245323 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:36.245294 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:36.253520 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:36.253488 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:40:36.954786 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:36.954751 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:40:37.121719 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:37.121683 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:40:37.223866 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:37.223772 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:40:49.803367 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:49.803319 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:40:49.904816 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:49.904769 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:40:50.906620 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:50.906576 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bcb48b54d-f4nrj" podUID="ac8cd84d-9b88-44dd-a041-4824961180af" containerName="console" containerID="cri-o://c885cf38e7fb93f7e00f1379c3a4776b0513348bc3c927390d7d9d9fc889acf7" gracePeriod=15 Jul 14 21:40:51.150100 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.150070 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:51.213182 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.213093 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-service-ca\") pod \"ac8cd84d-9b88-44dd-a041-4824961180af\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " Jul 14 21:40:51.213182 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.213131 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chq7n\" (UniqueName: \"kubernetes.io/projected/ac8cd84d-9b88-44dd-a041-4824961180af-kube-api-access-chq7n\") pod \"ac8cd84d-9b88-44dd-a041-4824961180af\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " Jul 14 21:40:51.213182 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.213150 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-console-config\") pod \"ac8cd84d-9b88-44dd-a041-4824961180af\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " Jul 14 21:40:51.213182 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.213176 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-trusted-ca-bundle\") pod \"ac8cd84d-9b88-44dd-a041-4824961180af\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " Jul 14 21:40:51.213538 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.213218 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-oauth-serving-cert\") pod \"ac8cd84d-9b88-44dd-a041-4824961180af\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " Jul 14 21:40:51.213538 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.213239 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac8cd84d-9b88-44dd-a041-4824961180af-console-serving-cert\") pod \"ac8cd84d-9b88-44dd-a041-4824961180af\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " Jul 14 21:40:51.213538 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.213285 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac8cd84d-9b88-44dd-a041-4824961180af-console-oauth-config\") pod \"ac8cd84d-9b88-44dd-a041-4824961180af\" (UID: \"ac8cd84d-9b88-44dd-a041-4824961180af\") " Jul 14 21:40:51.213703 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.213677 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-service-ca" (OuterVolumeSpecName: "service-ca") pod "ac8cd84d-9b88-44dd-a041-4824961180af" (UID: "ac8cd84d-9b88-44dd-a041-4824961180af"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:40:51.213790 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.213751 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ac8cd84d-9b88-44dd-a041-4824961180af" (UID: "ac8cd84d-9b88-44dd-a041-4824961180af"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:40:51.213893 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.213789 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-console-config" (OuterVolumeSpecName: "console-config") pod "ac8cd84d-9b88-44dd-a041-4824961180af" (UID: "ac8cd84d-9b88-44dd-a041-4824961180af"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:40:51.213893 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.213756 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ac8cd84d-9b88-44dd-a041-4824961180af" (UID: "ac8cd84d-9b88-44dd-a041-4824961180af"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:40:51.215587 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.215552 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac8cd84d-9b88-44dd-a041-4824961180af-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ac8cd84d-9b88-44dd-a041-4824961180af" (UID: "ac8cd84d-9b88-44dd-a041-4824961180af"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:40:51.215587 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.215571 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8cd84d-9b88-44dd-a041-4824961180af-kube-api-access-chq7n" (OuterVolumeSpecName: "kube-api-access-chq7n") pod "ac8cd84d-9b88-44dd-a041-4824961180af" (UID: "ac8cd84d-9b88-44dd-a041-4824961180af"). InnerVolumeSpecName "kube-api-access-chq7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:40:51.215711 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.215632 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac8cd84d-9b88-44dd-a041-4824961180af-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ac8cd84d-9b88-44dd-a041-4824961180af" (UID: "ac8cd84d-9b88-44dd-a041-4824961180af"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:40:51.314152 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.314110 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chq7n\" (UniqueName: \"kubernetes.io/projected/ac8cd84d-9b88-44dd-a041-4824961180af-kube-api-access-chq7n\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:40:51.314152 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.314143 2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-console-config\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:40:51.314152 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.314153 2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-trusted-ca-bundle\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:40:51.314152 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.314163 2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-oauth-serving-cert\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:40:51.314443 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.314171 2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac8cd84d-9b88-44dd-a041-4824961180af-console-serving-cert\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:40:51.314443 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.314180 2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac8cd84d-9b88-44dd-a041-4824961180af-console-oauth-config\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:40:51.314443 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:51.314213 2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac8cd84d-9b88-44dd-a041-4824961180af-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:40:52.013565 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:52.013527 2552 generic.go:356] "Generic (PLEG): container finished" podID="ac8cd84d-9b88-44dd-a041-4824961180af" containerID="c885cf38e7fb93f7e00f1379c3a4776b0513348bc3c927390d7d9d9fc889acf7" exitCode=0 Jul 14 21:40:52.013918 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:52.013595 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcb48b54d-f4nrj" Jul 14 21:40:52.013918 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:52.013619 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcb48b54d-f4nrj" event={"ID":"ac8cd84d-9b88-44dd-a041-4824961180af","Type":"ContainerDied","Data":"c885cf38e7fb93f7e00f1379c3a4776b0513348bc3c927390d7d9d9fc889acf7"} Jul 14 21:40:52.013918 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:52.013694 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcb48b54d-f4nrj" event={"ID":"ac8cd84d-9b88-44dd-a041-4824961180af","Type":"ContainerDied","Data":"dbb4d23c195d947c4d97558572837f0e7b1a99053bf389dc00572aed6541f319"} Jul 14 21:40:52.013918 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:52.013716 2552 scope.go:117] "RemoveContainer" containerID="c885cf38e7fb93f7e00f1379c3a4776b0513348bc3c927390d7d9d9fc889acf7" Jul 14 21:40:52.022001 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:52.021980 2552 scope.go:117] "RemoveContainer" containerID="c885cf38e7fb93f7e00f1379c3a4776b0513348bc3c927390d7d9d9fc889acf7" Jul 14 21:40:52.022277 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:40:52.022259 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c885cf38e7fb93f7e00f1379c3a4776b0513348bc3c927390d7d9d9fc889acf7\": container with ID starting with c885cf38e7fb93f7e00f1379c3a4776b0513348bc3c927390d7d9d9fc889acf7 not found: ID does not exist" containerID="c885cf38e7fb93f7e00f1379c3a4776b0513348bc3c927390d7d9d9fc889acf7" Jul 14 21:40:52.022327 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:52.022285 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c885cf38e7fb93f7e00f1379c3a4776b0513348bc3c927390d7d9d9fc889acf7"} err="failed to get container status \"c885cf38e7fb93f7e00f1379c3a4776b0513348bc3c927390d7d9d9fc889acf7\": rpc error: code = NotFound desc = could not find container \"c885cf38e7fb93f7e00f1379c3a4776b0513348bc3c927390d7d9d9fc889acf7\": container with ID starting with c885cf38e7fb93f7e00f1379c3a4776b0513348bc3c927390d7d9d9fc889acf7 not found: ID does not exist" Jul 14 21:40:52.031850 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:52.031825 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bcb48b54d-f4nrj"] Jul 14 21:40:52.038042 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:52.038017 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bcb48b54d-f4nrj"] Jul 14 21:40:53.990900 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:53.990865 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac8cd84d-9b88-44dd-a041-4824961180af" path="/var/lib/kubelet/pods/ac8cd84d-9b88-44dd-a041-4824961180af/volumes" Jul 14 21:40:56.810341 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:56.810310 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:40:56.877767 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:40:56.877729 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:41:17.152916 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:17.152878 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:41:17.250724 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:17.250678 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:41:17.793603 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:17.793568 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg"] Jul 14 21:41:17.793857 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:17.793845 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac8cd84d-9b88-44dd-a041-4824961180af" containerName="console" Jul 14 21:41:17.793900 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:17.793858 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac8cd84d-9b88-44dd-a041-4824961180af" containerName="console" Jul 14 21:41:17.793933 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:17.793920 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac8cd84d-9b88-44dd-a041-4824961180af" containerName="console" Jul 14 21:41:17.795880 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:17.795862 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" Jul 14 21:41:17.798748 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:17.798724 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 21:41:17.799766 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:17.799748 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 21:41:17.799836 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:17.799748 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-97wcb\"" Jul 14 21:41:17.810784 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:17.810755 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg"] Jul 14 21:41:17.924016 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:17.923972 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39a1377b-d475-4a2b-82f7-de1ae8802d41-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg\" (UID: \"39a1377b-d475-4a2b-82f7-de1ae8802d41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" Jul 14 21:41:17.924016 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:17.924019 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f69t\" (UniqueName: \"kubernetes.io/projected/39a1377b-d475-4a2b-82f7-de1ae8802d41-kube-api-access-7f69t\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg\" (UID: \"39a1377b-d475-4a2b-82f7-de1ae8802d41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" Jul 14 21:41:17.924306 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:17.924129 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39a1377b-d475-4a2b-82f7-de1ae8802d41-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg\" (UID: \"39a1377b-d475-4a2b-82f7-de1ae8802d41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" Jul 14 21:41:18.025349 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:18.025310 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39a1377b-d475-4a2b-82f7-de1ae8802d41-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg\" (UID: \"39a1377b-d475-4a2b-82f7-de1ae8802d41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" Jul 14 21:41:18.025521 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:18.025390 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39a1377b-d475-4a2b-82f7-de1ae8802d41-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg\" (UID: \"39a1377b-d475-4a2b-82f7-de1ae8802d41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" Jul 14 21:41:18.025521 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:18.025421 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f69t\" (UniqueName: \"kubernetes.io/projected/39a1377b-d475-4a2b-82f7-de1ae8802d41-kube-api-access-7f69t\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg\" (UID: \"39a1377b-d475-4a2b-82f7-de1ae8802d41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" Jul 14 21:41:18.025772 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:18.025749 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39a1377b-d475-4a2b-82f7-de1ae8802d41-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg\" (UID: \"39a1377b-d475-4a2b-82f7-de1ae8802d41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" Jul 14 21:41:18.025809 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:18.025763 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39a1377b-d475-4a2b-82f7-de1ae8802d41-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg\" (UID: \"39a1377b-d475-4a2b-82f7-de1ae8802d41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" Jul 14 21:41:18.036016 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:18.035982 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f69t\" (UniqueName: \"kubernetes.io/projected/39a1377b-d475-4a2b-82f7-de1ae8802d41-kube-api-access-7f69t\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg\" (UID: \"39a1377b-d475-4a2b-82f7-de1ae8802d41\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" Jul 14 21:41:18.105033 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:18.104995 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" Jul 14 21:41:18.237931 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:18.237901 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg"] Jul 14 21:41:18.240293 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:41:18.240257 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a1377b_d475_4a2b_82f7_de1ae8802d41.slice/crio-8c36bd98de0facd2e15c23157b7ed3a6a6fd847ef357b29fc3c6777659e99f96 WatchSource:0}: Error finding container 8c36bd98de0facd2e15c23157b7ed3a6a6fd847ef357b29fc3c6777659e99f96: Status 404 returned error can't find the container with id 8c36bd98de0facd2e15c23157b7ed3a6a6fd847ef357b29fc3c6777659e99f96 Jul 14 21:41:19.124849 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:19.124797 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" event={"ID":"39a1377b-d475-4a2b-82f7-de1ae8802d41","Type":"ContainerStarted","Data":"8c36bd98de0facd2e15c23157b7ed3a6a6fd847ef357b29fc3c6777659e99f96"} Jul 14 21:41:25.151008 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:25.150968 2552 generic.go:356] "Generic (PLEG): container finished" podID="39a1377b-d475-4a2b-82f7-de1ae8802d41" containerID="b7d7aac5e3a06205f198d54edaa0633fe8d56400080309cbe23a4d2f9ee2128f" exitCode=0 Jul 14 21:41:25.151531 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:25.151051 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" event={"ID":"39a1377b-d475-4a2b-82f7-de1ae8802d41","Type":"ContainerDied","Data":"b7d7aac5e3a06205f198d54edaa0633fe8d56400080309cbe23a4d2f9ee2128f"} Jul 14 21:41:27.160623 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:27.160529 2552 generic.go:356] "Generic (PLEG): container finished" podID="39a1377b-d475-4a2b-82f7-de1ae8802d41" containerID="b4cd7683e990ea7fe1bdd810a9e3a9a54a88dd3b2867442c20b0360d8f19e1d9" exitCode=0 Jul 14 21:41:27.160623 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:27.160578 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" event={"ID":"39a1377b-d475-4a2b-82f7-de1ae8802d41","Type":"ContainerDied","Data":"b4cd7683e990ea7fe1bdd810a9e3a9a54a88dd3b2867442c20b0360d8f19e1d9"} Jul 14 21:41:35.196575 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:35.196536 2552 generic.go:356] "Generic (PLEG): container finished" podID="39a1377b-d475-4a2b-82f7-de1ae8802d41" containerID="df55dd959893b7904b8c3bc0a0169c0e646aca65bd7ec9d7ff9d3c2a27001fb3" exitCode=0 Jul 14 21:41:35.196996 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:35.196611 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" event={"ID":"39a1377b-d475-4a2b-82f7-de1ae8802d41","Type":"ContainerDied","Data":"df55dd959893b7904b8c3bc0a0169c0e646aca65bd7ec9d7ff9d3c2a27001fb3"} Jul 14 21:41:36.324790 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:36.324762 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" Jul 14 21:41:36.480755 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:36.480659 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39a1377b-d475-4a2b-82f7-de1ae8802d41-bundle\") pod \"39a1377b-d475-4a2b-82f7-de1ae8802d41\" (UID: \"39a1377b-d475-4a2b-82f7-de1ae8802d41\") " Jul 14 21:41:36.480755 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:36.480698 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39a1377b-d475-4a2b-82f7-de1ae8802d41-util\") pod \"39a1377b-d475-4a2b-82f7-de1ae8802d41\" (UID: \"39a1377b-d475-4a2b-82f7-de1ae8802d41\") " Jul 14 21:41:36.480755 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:36.480748 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f69t\" (UniqueName: \"kubernetes.io/projected/39a1377b-d475-4a2b-82f7-de1ae8802d41-kube-api-access-7f69t\") pod \"39a1377b-d475-4a2b-82f7-de1ae8802d41\" (UID: \"39a1377b-d475-4a2b-82f7-de1ae8802d41\") " Jul 14 21:41:36.481484 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:36.481454 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a1377b-d475-4a2b-82f7-de1ae8802d41-bundle" (OuterVolumeSpecName: "bundle") pod "39a1377b-d475-4a2b-82f7-de1ae8802d41" (UID: "39a1377b-d475-4a2b-82f7-de1ae8802d41"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:41:36.483043 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:36.483017 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a1377b-d475-4a2b-82f7-de1ae8802d41-kube-api-access-7f69t" (OuterVolumeSpecName: "kube-api-access-7f69t") pod "39a1377b-d475-4a2b-82f7-de1ae8802d41" (UID: "39a1377b-d475-4a2b-82f7-de1ae8802d41"). InnerVolumeSpecName "kube-api-access-7f69t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:41:36.485706 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:36.485684 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a1377b-d475-4a2b-82f7-de1ae8802d41-util" (OuterVolumeSpecName: "util") pod "39a1377b-d475-4a2b-82f7-de1ae8802d41" (UID: "39a1377b-d475-4a2b-82f7-de1ae8802d41"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:41:36.582063 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:36.582029 2552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39a1377b-d475-4a2b-82f7-de1ae8802d41-bundle\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:41:36.582063 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:36.582056 2552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39a1377b-d475-4a2b-82f7-de1ae8802d41-util\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:41:36.582063 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:36.582065 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f69t\" (UniqueName: \"kubernetes.io/projected/39a1377b-d475-4a2b-82f7-de1ae8802d41-kube-api-access-7f69t\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:41:37.206857 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:37.206828 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" Jul 14 21:41:37.206857 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:37.206843 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540phmrg" event={"ID":"39a1377b-d475-4a2b-82f7-de1ae8802d41","Type":"ContainerDied","Data":"8c36bd98de0facd2e15c23157b7ed3a6a6fd847ef357b29fc3c6777659e99f96"} Jul 14 21:41:37.207067 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:37.206886 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c36bd98de0facd2e15c23157b7ed3a6a6fd847ef357b29fc3c6777659e99f96" Jul 14 21:41:38.795527 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:38.795494 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:41:39.773441 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:39.773399 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:41:40.873219 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:40.873169 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlb5x"] Jul 14 21:41:40.873713 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:40.873462 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39a1377b-d475-4a2b-82f7-de1ae8802d41" containerName="pull" Jul 14 21:41:40.873713 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:40.873473 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="39a1377b-d475-4a2b-82f7-de1ae8802d41" containerName="pull" Jul 14 21:41:40.873713 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:40.873480 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39a1377b-d475-4a2b-82f7-de1ae8802d41" containerName="extract" Jul 14 21:41:40.873713 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:40.873486 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="39a1377b-d475-4a2b-82f7-de1ae8802d41" containerName="extract" Jul 14 21:41:40.873713 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:40.873493 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39a1377b-d475-4a2b-82f7-de1ae8802d41" containerName="util" Jul 14 21:41:40.873713 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:40.873499 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="39a1377b-d475-4a2b-82f7-de1ae8802d41" containerName="util" Jul 14 21:41:40.873713 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:40.873557 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="39a1377b-d475-4a2b-82f7-de1ae8802d41" containerName="extract" Jul 14 21:41:40.907091 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:40.907060 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlb5x"] Jul 14 21:41:40.907268 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:40.907179 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlb5x" Jul 14 21:41:40.910256 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:40.910226 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7zsls\"" Jul 14 21:41:40.910651 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:40.910633 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 14 21:41:40.910651 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:40.910646 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 14 21:41:41.019779 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:41.019731 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95f157d4-47ef-4aca-b06c-815c4b0e0542-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlb5x\" (UID: \"95f157d4-47ef-4aca-b06c-815c4b0e0542\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlb5x" Jul 14 21:41:41.019954 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:41.019788 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxtbb\" (UniqueName: \"kubernetes.io/projected/95f157d4-47ef-4aca-b06c-815c4b0e0542-kube-api-access-bxtbb\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlb5x\" (UID: \"95f157d4-47ef-4aca-b06c-815c4b0e0542\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlb5x" Jul 14 21:41:41.121016 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:41.120966 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95f157d4-47ef-4aca-b06c-815c4b0e0542-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlb5x\" (UID: \"95f157d4-47ef-4aca-b06c-815c4b0e0542\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlb5x" Jul 14 21:41:41.121016 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:41.121015 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxtbb\" (UniqueName: \"kubernetes.io/projected/95f157d4-47ef-4aca-b06c-815c4b0e0542-kube-api-access-bxtbb\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlb5x\" (UID: \"95f157d4-47ef-4aca-b06c-815c4b0e0542\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlb5x" Jul 14 21:41:41.121384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:41.121365 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95f157d4-47ef-4aca-b06c-815c4b0e0542-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlb5x\" (UID: \"95f157d4-47ef-4aca-b06c-815c4b0e0542\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlb5x" Jul 14 21:41:41.141073 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:41.140989 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxtbb\" (UniqueName: \"kubernetes.io/projected/95f157d4-47ef-4aca-b06c-815c4b0e0542-kube-api-access-bxtbb\") pod \"cert-manager-operator-controller-manager-55c444bf86-dlb5x\" (UID: \"95f157d4-47ef-4aca-b06c-815c4b0e0542\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlb5x" Jul 14 21:41:41.216271 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:41.216235 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlb5x" Jul 14 21:41:41.386064 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:41.386036 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlb5x"] Jul 14 21:41:41.389076 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:41:41.389044 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f157d4_47ef_4aca_b06c_815c4b0e0542.slice/crio-5fc8d588123f3c1f00f21e2d20ef71cf447269ff07300eeeb24c4594ea3a3f91 WatchSource:0}: Error finding container 5fc8d588123f3c1f00f21e2d20ef71cf447269ff07300eeeb24c4594ea3a3f91: Status 404 returned error can't find the container with id 5fc8d588123f3c1f00f21e2d20ef71cf447269ff07300eeeb24c4594ea3a3f91 Jul 14 21:41:42.227650 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:42.227603 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlb5x" event={"ID":"95f157d4-47ef-4aca-b06c-815c4b0e0542","Type":"ContainerStarted","Data":"5fc8d588123f3c1f00f21e2d20ef71cf447269ff07300eeeb24c4594ea3a3f91"} Jul 14 21:41:42.434642 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:42.434606 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:41:42.541871 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:42.541790 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:41:44.217133 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:44.217102 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:41:44.238205 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:44.238167 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlb5x" event={"ID":"95f157d4-47ef-4aca-b06c-815c4b0e0542","Type":"ContainerStarted","Data":"7e98ebdd015f681ae39980294ccc2b7a3b607693b8f39420f3355079f9e8f13d"} Jul 14 21:41:44.265010 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:44.264949 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dlb5x" podStartSLOduration=2.108961532 podStartE2EDuration="4.26493271s" totalImagesPullingTime="2.155971178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:41:44.259175063 +0000 UTC m=+270.797880735" watchObservedRunningTime="2026-07-14 21:41:44.26493271 +0000 UTC m=+270.803638383" Jul 14 21:41:44.316092 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:44.316055 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:41:44.644118 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:44.644078 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:41:44.776508 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:44.776467 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:41:45.311384 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:45.311349 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj"] Jul 14 21:41:45.313578 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:45.313560 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" Jul 14 21:41:45.316324 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:45.316293 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 21:41:45.316455 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:45.316360 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 21:41:45.317379 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:45.317361 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-97wcb\"" Jul 14 21:41:45.324709 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:45.324689 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj"] Jul 14 21:41:45.457956 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:45.457924 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj\" (UID: \"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" Jul 14 21:41:45.457956 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:45.457964 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccsgb\" (UniqueName: \"kubernetes.io/projected/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-kube-api-access-ccsgb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj\" (UID: \"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" Jul 14 21:41:45.458170 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:45.458042 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj\" (UID: \"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" Jul 14 21:41:45.558716 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:45.558677 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj\" (UID: \"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" Jul 14 21:41:45.558716 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:45.558720 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccsgb\" (UniqueName: \"kubernetes.io/projected/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-kube-api-access-ccsgb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj\" (UID: \"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" Jul 14 21:41:45.558923 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:45.558751 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj\" (UID: \"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" Jul 14 21:41:45.559097 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:45.559075 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj\" (UID: \"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" Jul 14 21:41:45.559147 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:45.559129 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj\" (UID: \"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" Jul 14 21:41:45.568979 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:45.568908 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccsgb\" (UniqueName: \"kubernetes.io/projected/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-kube-api-access-ccsgb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj\" (UID: \"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" Jul 14 21:41:45.622553 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:45.622518 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" Jul 14 21:41:45.762806 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:45.762733 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj"] Jul 14 21:41:45.764863 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:41:45.764827 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0520904a_bc0c_4cd3_a5d2_7e860e83b0b9.slice/crio-c7c1ce2a199b60cc378b4eb3d652cbfd3cf83c7de927b76e2e8fcb619cc558a2 WatchSource:0}: Error finding container c7c1ce2a199b60cc378b4eb3d652cbfd3cf83c7de927b76e2e8fcb619cc558a2: Status 404 returned error can't find the container with id c7c1ce2a199b60cc378b4eb3d652cbfd3cf83c7de927b76e2e8fcb619cc558a2 Jul 14 21:41:46.247619 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:46.247586 2552 generic.go:356] "Generic (PLEG): container finished" podID="0520904a-bc0c-4cd3-a5d2-7e860e83b0b9" containerID="03bfe1c15e02715ccbb474419cc7ebd36b8051c4006eef6f536b03b6ebd703b7" exitCode=0 Jul 14 21:41:46.247787 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:46.247665 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" event={"ID":"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9","Type":"ContainerDied","Data":"03bfe1c15e02715ccbb474419cc7ebd36b8051c4006eef6f536b03b6ebd703b7"} Jul 14 21:41:46.247787 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:46.247715 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" event={"ID":"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9","Type":"ContainerStarted","Data":"c7c1ce2a199b60cc378b4eb3d652cbfd3cf83c7de927b76e2e8fcb619cc558a2"} Jul 14 21:41:46.433226 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:46.433168 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:41:46.625065 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:46.625025 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:41:49.403809 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:49.403773 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:41:49.467920 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:49.467885 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:41:50.266487 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:50.266446 2552 generic.go:356] "Generic (PLEG): container finished" podID="0520904a-bc0c-4cd3-a5d2-7e860e83b0b9" containerID="ec9ba685540ba038dc681b5ac87fd2465d2aec1a6a89ed736237fd8b6c0f185e" exitCode=0 Jul 14 21:41:50.266664 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:50.266539 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" event={"ID":"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9","Type":"ContainerDied","Data":"ec9ba685540ba038dc681b5ac87fd2465d2aec1a6a89ed736237fd8b6c0f185e"} Jul 14 21:41:51.272946 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:51.272903 2552 generic.go:356] "Generic (PLEG): container finished" podID="0520904a-bc0c-4cd3-a5d2-7e860e83b0b9" containerID="35fc30339759c5d8bb3601937e772b40ce07d7c9d7587444556a0ff86ac2ac19" exitCode=0 Jul 14 21:41:51.273349 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:51.272988 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" event={"ID":"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9","Type":"ContainerDied","Data":"35fc30339759c5d8bb3601937e772b40ce07d7c9d7587444556a0ff86ac2ac19"} Jul 14 21:41:52.405275 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:52.405248 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" Jul 14 21:41:52.424023 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:52.423994 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-util\") pod \"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9\" (UID: \"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9\") " Jul 14 21:41:52.424145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:52.424031 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccsgb\" (UniqueName: \"kubernetes.io/projected/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-kube-api-access-ccsgb\") pod \"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9\" (UID: \"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9\") " Jul 14 21:41:52.424145 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:52.424102 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-bundle\") pod \"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9\" (UID: \"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9\") " Jul 14 21:41:52.424527 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:52.424494 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-bundle" (OuterVolumeSpecName: "bundle") pod "0520904a-bc0c-4cd3-a5d2-7e860e83b0b9" (UID: "0520904a-bc0c-4cd3-a5d2-7e860e83b0b9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:41:52.426230 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:52.426204 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-kube-api-access-ccsgb" (OuterVolumeSpecName: "kube-api-access-ccsgb") pod "0520904a-bc0c-4cd3-a5d2-7e860e83b0b9" (UID: "0520904a-bc0c-4cd3-a5d2-7e860e83b0b9"). InnerVolumeSpecName "kube-api-access-ccsgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:41:52.428531 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:52.428511 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-util" (OuterVolumeSpecName: "util") pod "0520904a-bc0c-4cd3-a5d2-7e860e83b0b9" (UID: "0520904a-bc0c-4cd3-a5d2-7e860e83b0b9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:41:52.525406 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:52.525365 2552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-bundle\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:41:52.525406 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:52.525398 2552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-util\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:41:52.525406 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:52.525410 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccsgb\" (UniqueName: \"kubernetes.io/projected/0520904a-bc0c-4cd3-a5d2-7e860e83b0b9-kube-api-access-ccsgb\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:41:53.284053 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:53.284023 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" Jul 14 21:41:53.284053 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:53.284035 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrjsj" event={"ID":"0520904a-bc0c-4cd3-a5d2-7e860e83b0b9","Type":"ContainerDied","Data":"c7c1ce2a199b60cc378b4eb3d652cbfd3cf83c7de927b76e2e8fcb619cc558a2"} Jul 14 21:41:53.284288 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:53.284081 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7c1ce2a199b60cc378b4eb3d652cbfd3cf83c7de927b76e2e8fcb619cc558a2" Jul 14 21:41:54.814320 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.814288 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dcpkt"] Jul 14 21:41:54.814681 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.814580 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0520904a-bc0c-4cd3-a5d2-7e860e83b0b9" containerName="util" Jul 14 21:41:54.814681 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.814590 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0520904a-bc0c-4cd3-a5d2-7e860e83b0b9" containerName="util" Jul 14 21:41:54.814681 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.814602 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0520904a-bc0c-4cd3-a5d2-7e860e83b0b9" containerName="pull" Jul 14 21:41:54.814681 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.814607 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0520904a-bc0c-4cd3-a5d2-7e860e83b0b9" containerName="pull" Jul 14 21:41:54.814681 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.814619 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0520904a-bc0c-4cd3-a5d2-7e860e83b0b9" containerName="extract" Jul 14 21:41:54.814681 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.814624 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0520904a-bc0c-4cd3-a5d2-7e860e83b0b9" containerName="extract" Jul 14 21:41:54.814681 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.814672 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="0520904a-bc0c-4cd3-a5d2-7e860e83b0b9" containerName="extract" Jul 14 21:41:54.818400 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.818381 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dcpkt" Jul 14 21:41:54.821270 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.821246 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 14 21:41:54.821391 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.821246 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 14 21:41:54.821391 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.821248 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7j5v7\"" Jul 14 21:41:54.828400 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.828378 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dcpkt"] Jul 14 21:41:54.837204 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.837165 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:41:54.840121 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.840101 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b7cbc72-952f-48dd-b954-9da0231d0817-bound-sa-token\") pod \"cert-manager-55b4d954d5-dcpkt\" (UID: \"7b7cbc72-952f-48dd-b954-9da0231d0817\") " pod="cert-manager/cert-manager-55b4d954d5-dcpkt" Jul 14 21:41:54.840278 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.840148 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttz46\" (UniqueName: \"kubernetes.io/projected/7b7cbc72-952f-48dd-b954-9da0231d0817-kube-api-access-ttz46\") pod \"cert-manager-55b4d954d5-dcpkt\" (UID: \"7b7cbc72-952f-48dd-b954-9da0231d0817\") " pod="cert-manager/cert-manager-55b4d954d5-dcpkt" Jul 14 21:41:54.941322 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.941287 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b7cbc72-952f-48dd-b954-9da0231d0817-bound-sa-token\") pod \"cert-manager-55b4d954d5-dcpkt\" (UID: \"7b7cbc72-952f-48dd-b954-9da0231d0817\") " pod="cert-manager/cert-manager-55b4d954d5-dcpkt" Jul 14 21:41:54.941488 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.941339 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttz46\" (UniqueName: \"kubernetes.io/projected/7b7cbc72-952f-48dd-b954-9da0231d0817-kube-api-access-ttz46\") pod \"cert-manager-55b4d954d5-dcpkt\" (UID: \"7b7cbc72-952f-48dd-b954-9da0231d0817\") " pod="cert-manager/cert-manager-55b4d954d5-dcpkt" Jul 14 21:41:54.950884 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.950855 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttz46\" (UniqueName: \"kubernetes.io/projected/7b7cbc72-952f-48dd-b954-9da0231d0817-kube-api-access-ttz46\") pod \"cert-manager-55b4d954d5-dcpkt\" (UID: \"7b7cbc72-952f-48dd-b954-9da0231d0817\") " pod="cert-manager/cert-manager-55b4d954d5-dcpkt" Jul 14 21:41:54.951017 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:54.950857 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b7cbc72-952f-48dd-b954-9da0231d0817-bound-sa-token\") pod \"cert-manager-55b4d954d5-dcpkt\" (UID: \"7b7cbc72-952f-48dd-b954-9da0231d0817\") " pod="cert-manager/cert-manager-55b4d954d5-dcpkt" Jul 14 21:41:55.135810 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:55.135776 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dcpkt" Jul 14 21:41:55.270448 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:55.270422 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dcpkt"] Jul 14 21:41:55.272958 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:41:55.272924 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b7cbc72_952f_48dd_b954_9da0231d0817.slice/crio-e90c54b7fed918639c6aa8f557334c6c2c74e7aa4f28171739bc12d2375d3ca7 WatchSource:0}: Error finding container e90c54b7fed918639c6aa8f557334c6c2c74e7aa4f28171739bc12d2375d3ca7: Status 404 returned error can't find the container with id e90c54b7fed918639c6aa8f557334c6c2c74e7aa4f28171739bc12d2375d3ca7 Jul 14 21:41:55.298261 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:55.298230 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dcpkt" event={"ID":"7b7cbc72-952f-48dd-b954-9da0231d0817","Type":"ContainerStarted","Data":"e90c54b7fed918639c6aa8f557334c6c2c74e7aa4f28171739bc12d2375d3ca7"} Jul 14 21:41:58.313701 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:58.313661 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dcpkt" event={"ID":"7b7cbc72-952f-48dd-b954-9da0231d0817","Type":"ContainerStarted","Data":"41956411f44e82f3f3a74993e3826fcea244e7d3c3e4c3ce75743cc39d637cde"} Jul 14 21:41:58.333734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:58.333675 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-dcpkt" podStartSLOduration=1.864743601 podStartE2EDuration="4.333657303s" totalImagesPullingTime="2.468913702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:41:58.332017178 +0000 UTC m=+284.870722851" watchObservedRunningTime="2026-07-14 21:41:58.333657303 +0000 UTC m=+284.872362975" Jul 14 21:41:58.824492 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:58.824460 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:41:58.928660 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:41:58.928611 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:04.663184 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:04.663145 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:04.764100 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:04.764057 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:06.285218 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.285159 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7"] Jul 14 21:42:06.287957 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.287930 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" Jul 14 21:42:06.290677 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.290653 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 21:42:06.290786 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.290651 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 21:42:06.291707 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.291685 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-97wcb\"" Jul 14 21:42:06.296867 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.296843 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7"] Jul 14 21:42:06.336424 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.336389 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2178c11f-55af-4d4b-93a6-de5a19db13b7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7\" (UID: \"2178c11f-55af-4d4b-93a6-de5a19db13b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" Jul 14 21:42:06.336597 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.336447 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2178c11f-55af-4d4b-93a6-de5a19db13b7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7\" (UID: \"2178c11f-55af-4d4b-93a6-de5a19db13b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" Jul 14 21:42:06.336597 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.336515 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rfj8\" (UniqueName: \"kubernetes.io/projected/2178c11f-55af-4d4b-93a6-de5a19db13b7-kube-api-access-8rfj8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7\" (UID: \"2178c11f-55af-4d4b-93a6-de5a19db13b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" Jul 14 21:42:06.437371 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.437332 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rfj8\" (UniqueName: \"kubernetes.io/projected/2178c11f-55af-4d4b-93a6-de5a19db13b7-kube-api-access-8rfj8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7\" (UID: \"2178c11f-55af-4d4b-93a6-de5a19db13b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" Jul 14 21:42:06.437568 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.437398 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2178c11f-55af-4d4b-93a6-de5a19db13b7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7\" (UID: \"2178c11f-55af-4d4b-93a6-de5a19db13b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" Jul 14 21:42:06.437568 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.437439 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2178c11f-55af-4d4b-93a6-de5a19db13b7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7\" (UID: \"2178c11f-55af-4d4b-93a6-de5a19db13b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" Jul 14 21:42:06.437788 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.437772 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2178c11f-55af-4d4b-93a6-de5a19db13b7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7\" (UID: \"2178c11f-55af-4d4b-93a6-de5a19db13b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" Jul 14 21:42:06.437823 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.437803 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2178c11f-55af-4d4b-93a6-de5a19db13b7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7\" (UID: \"2178c11f-55af-4d4b-93a6-de5a19db13b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" Jul 14 21:42:06.447326 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.447302 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rfj8\" (UniqueName: \"kubernetes.io/projected/2178c11f-55af-4d4b-93a6-de5a19db13b7-kube-api-access-8rfj8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7\" (UID: \"2178c11f-55af-4d4b-93a6-de5a19db13b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" Jul 14 21:42:06.576599 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.576511 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:06.598074 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.598047 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" Jul 14 21:42:06.730427 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:06.730399 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7"] Jul 14 21:42:06.732771 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:42:06.732742 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2178c11f_55af_4d4b_93a6_de5a19db13b7.slice/crio-fd24c313c3c2f7b1d47cb80731714fdaf2af1dd7deb83667578b5284b5912b12 WatchSource:0}: Error finding container fd24c313c3c2f7b1d47cb80731714fdaf2af1dd7deb83667578b5284b5912b12: Status 404 returned error can't find the container with id fd24c313c3c2f7b1d47cb80731714fdaf2af1dd7deb83667578b5284b5912b12 Jul 14 21:42:07.353580 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:07.353549 2552 generic.go:356] "Generic (PLEG): container finished" podID="2178c11f-55af-4d4b-93a6-de5a19db13b7" containerID="872efd442275cb64a4f97358d17bbcada4962cb8538a8dad2cd5d8136eff5748" exitCode=0 Jul 14 21:42:07.353989 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:07.353640 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" event={"ID":"2178c11f-55af-4d4b-93a6-de5a19db13b7","Type":"ContainerDied","Data":"872efd442275cb64a4f97358d17bbcada4962cb8538a8dad2cd5d8136eff5748"} Jul 14 21:42:07.353989 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:07.353694 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" event={"ID":"2178c11f-55af-4d4b-93a6-de5a19db13b7","Type":"ContainerStarted","Data":"fd24c313c3c2f7b1d47cb80731714fdaf2af1dd7deb83667578b5284b5912b12"} Jul 14 21:42:08.359743 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:08.359710 2552 generic.go:356] "Generic (PLEG): container finished" podID="2178c11f-55af-4d4b-93a6-de5a19db13b7" containerID="396ac02f0dbdab0a226294b70854f648d1417ef718c3ae1d623edd41aef49c2e" exitCode=0 Jul 14 21:42:08.360272 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:08.359775 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" event={"ID":"2178c11f-55af-4d4b-93a6-de5a19db13b7","Type":"ContainerDied","Data":"396ac02f0dbdab0a226294b70854f648d1417ef718c3ae1d623edd41aef49c2e"} Jul 14 21:42:09.366846 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:09.366805 2552 generic.go:356] "Generic (PLEG): container finished" podID="2178c11f-55af-4d4b-93a6-de5a19db13b7" containerID="b4f24d9efbb04aa9cf7239426087e33b4cd6d398ff559aebe7badb53dc962aca" exitCode=0 Jul 14 21:42:09.367361 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:09.366844 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" event={"ID":"2178c11f-55af-4d4b-93a6-de5a19db13b7","Type":"ContainerDied","Data":"b4f24d9efbb04aa9cf7239426087e33b4cd6d398ff559aebe7badb53dc962aca"} Jul 14 21:42:10.497077 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:10.497050 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" Jul 14 21:42:10.575212 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:10.575154 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2178c11f-55af-4d4b-93a6-de5a19db13b7-bundle\") pod \"2178c11f-55af-4d4b-93a6-de5a19db13b7\" (UID: \"2178c11f-55af-4d4b-93a6-de5a19db13b7\") " Jul 14 21:42:10.575385 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:10.575245 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2178c11f-55af-4d4b-93a6-de5a19db13b7-util\") pod \"2178c11f-55af-4d4b-93a6-de5a19db13b7\" (UID: \"2178c11f-55af-4d4b-93a6-de5a19db13b7\") " Jul 14 21:42:10.575385 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:10.575278 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rfj8\" (UniqueName: \"kubernetes.io/projected/2178c11f-55af-4d4b-93a6-de5a19db13b7-kube-api-access-8rfj8\") pod \"2178c11f-55af-4d4b-93a6-de5a19db13b7\" (UID: \"2178c11f-55af-4d4b-93a6-de5a19db13b7\") " Jul 14 21:42:10.575880 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:10.575855 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2178c11f-55af-4d4b-93a6-de5a19db13b7-bundle" (OuterVolumeSpecName: "bundle") pod "2178c11f-55af-4d4b-93a6-de5a19db13b7" (UID: "2178c11f-55af-4d4b-93a6-de5a19db13b7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:42:10.577348 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:10.577321 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2178c11f-55af-4d4b-93a6-de5a19db13b7-kube-api-access-8rfj8" (OuterVolumeSpecName: "kube-api-access-8rfj8") pod "2178c11f-55af-4d4b-93a6-de5a19db13b7" (UID: "2178c11f-55af-4d4b-93a6-de5a19db13b7"). InnerVolumeSpecName "kube-api-access-8rfj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:42:10.581025 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:10.580994 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2178c11f-55af-4d4b-93a6-de5a19db13b7-util" (OuterVolumeSpecName: "util") pod "2178c11f-55af-4d4b-93a6-de5a19db13b7" (UID: "2178c11f-55af-4d4b-93a6-de5a19db13b7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:42:10.676256 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:10.676133 2552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2178c11f-55af-4d4b-93a6-de5a19db13b7-bundle\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:42:10.676256 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:10.676167 2552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2178c11f-55af-4d4b-93a6-de5a19db13b7-util\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:42:10.676256 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:10.676180 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rfj8\" (UniqueName: \"kubernetes.io/projected/2178c11f-55af-4d4b-93a6-de5a19db13b7-kube-api-access-8rfj8\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:42:11.377685 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:11.377639 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" event={"ID":"2178c11f-55af-4d4b-93a6-de5a19db13b7","Type":"ContainerDied","Data":"fd24c313c3c2f7b1d47cb80731714fdaf2af1dd7deb83667578b5284b5912b12"} Jul 14 21:42:11.377685 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:11.377681 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79kv7" Jul 14 21:42:11.377905 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:11.377700 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd24c313c3c2f7b1d47cb80731714fdaf2af1dd7deb83667578b5284b5912b12" Jul 14 21:42:13.896293 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:13.896251 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 21:42:13.898204 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:13.898161 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 21:42:13.905004 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:13.904969 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 21:42:13.908287 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:13.908267 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 21:42:13.909430 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:13.909403 2552 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 21:42:17.481389 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:17.481344 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:17.584740 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:17.584709 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:18.803003 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:18.802964 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:18.899681 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:18.899641 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:19.419663 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.419624 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:19.701127 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.701035 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:19.804543 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.804503 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:19.832264 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.832233 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh"] Jul 14 21:42:19.832548 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.832535 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2178c11f-55af-4d4b-93a6-de5a19db13b7" containerName="pull" Jul 14 21:42:19.832593 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.832550 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2178c11f-55af-4d4b-93a6-de5a19db13b7" containerName="pull" Jul 14 21:42:19.832593 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.832570 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2178c11f-55af-4d4b-93a6-de5a19db13b7" containerName="extract" Jul 14 21:42:19.832593 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.832576 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2178c11f-55af-4d4b-93a6-de5a19db13b7" containerName="extract" Jul 14 21:42:19.832593 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.832584 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2178c11f-55af-4d4b-93a6-de5a19db13b7" containerName="util" Jul 14 21:42:19.832593 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.832590 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2178c11f-55af-4d4b-93a6-de5a19db13b7" containerName="util" Jul 14 21:42:19.832767 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.832662 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2178c11f-55af-4d4b-93a6-de5a19db13b7" containerName="extract" Jul 14 21:42:19.834547 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.834528 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:19.837297 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.837273 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zzk6s\"" Jul 14 21:42:19.837427 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.837312 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 21:42:19.837553 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.837534 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 21:42:19.837625 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.837534 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 21:42:19.837914 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.837897 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 21:42:19.847641 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.847616 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh"] Jul 14 21:42:19.952421 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.952331 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1319bf20-cc7d-474e-bd56-89e5dee487b7-manifests\") pod \"opendatahub-operator-controller-manager-c55cf7c4c-zkxmh\" (UID: \"1319bf20-cc7d-474e-bd56-89e5dee487b7\") " pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:19.952421 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.952387 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1319bf20-cc7d-474e-bd56-89e5dee487b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-c55cf7c4c-zkxmh\" (UID: \"1319bf20-cc7d-474e-bd56-89e5dee487b7\") " pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:19.952627 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.952448 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1319bf20-cc7d-474e-bd56-89e5dee487b7-tmp\") pod \"opendatahub-operator-controller-manager-c55cf7c4c-zkxmh\" (UID: \"1319bf20-cc7d-474e-bd56-89e5dee487b7\") " pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:19.952627 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.952484 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf6vc\" (UniqueName: \"kubernetes.io/projected/1319bf20-cc7d-474e-bd56-89e5dee487b7-kube-api-access-sf6vc\") pod \"opendatahub-operator-controller-manager-c55cf7c4c-zkxmh\" (UID: \"1319bf20-cc7d-474e-bd56-89e5dee487b7\") " pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:19.952627 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.952514 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1319bf20-cc7d-474e-bd56-89e5dee487b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c55cf7c4c-zkxmh\" (UID: \"1319bf20-cc7d-474e-bd56-89e5dee487b7\") " pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:19.959702 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:19.959662 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:20.053479 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.053444 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1319bf20-cc7d-474e-bd56-89e5dee487b7-tmp\") pod \"opendatahub-operator-controller-manager-c55cf7c4c-zkxmh\" (UID: \"1319bf20-cc7d-474e-bd56-89e5dee487b7\") " pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:20.053705 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.053495 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf6vc\" (UniqueName: \"kubernetes.io/projected/1319bf20-cc7d-474e-bd56-89e5dee487b7-kube-api-access-sf6vc\") pod \"opendatahub-operator-controller-manager-c55cf7c4c-zkxmh\" (UID: \"1319bf20-cc7d-474e-bd56-89e5dee487b7\") " pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:20.053705 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.053531 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1319bf20-cc7d-474e-bd56-89e5dee487b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c55cf7c4c-zkxmh\" (UID: \"1319bf20-cc7d-474e-bd56-89e5dee487b7\") " pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:20.053705 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.053556 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1319bf20-cc7d-474e-bd56-89e5dee487b7-manifests\") pod \"opendatahub-operator-controller-manager-c55cf7c4c-zkxmh\" (UID: \"1319bf20-cc7d-474e-bd56-89e5dee487b7\") " pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:20.053705 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.053645 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1319bf20-cc7d-474e-bd56-89e5dee487b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-c55cf7c4c-zkxmh\" (UID: \"1319bf20-cc7d-474e-bd56-89e5dee487b7\") " pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:20.053977 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.053950 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1319bf20-cc7d-474e-bd56-89e5dee487b7-manifests\") pod \"opendatahub-operator-controller-manager-c55cf7c4c-zkxmh\" (UID: \"1319bf20-cc7d-474e-bd56-89e5dee487b7\") " pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:20.055922 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.055890 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1319bf20-cc7d-474e-bd56-89e5dee487b7-tmp\") pod \"opendatahub-operator-controller-manager-c55cf7c4c-zkxmh\" (UID: \"1319bf20-cc7d-474e-bd56-89e5dee487b7\") " pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:20.056043 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.056023 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1319bf20-cc7d-474e-bd56-89e5dee487b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-c55cf7c4c-zkxmh\" (UID: \"1319bf20-cc7d-474e-bd56-89e5dee487b7\") " pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:20.056083 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.056039 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1319bf20-cc7d-474e-bd56-89e5dee487b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c55cf7c4c-zkxmh\" (UID: \"1319bf20-cc7d-474e-bd56-89e5dee487b7\") " pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:20.060512 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.060485 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:20.062944 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.062907 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf6vc\" (UniqueName: \"kubernetes.io/projected/1319bf20-cc7d-474e-bd56-89e5dee487b7-kube-api-access-sf6vc\") pod \"opendatahub-operator-controller-manager-c55cf7c4c-zkxmh\" (UID: \"1319bf20-cc7d-474e-bd56-89e5dee487b7\") " pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:20.144990 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.144954 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:20.163551 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.163517 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:20.262388 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.262355 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:20.300211 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.300130 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh"] Jul 14 21:42:20.304777 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:42:20.304734 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1319bf20_cc7d_474e_bd56_89e5dee487b7.slice/crio-eceabf4201433feeba54fd7ec6145bd5d5654fee5f8e42787a1cf39b3e80e7b1 WatchSource:0}: Error finding container eceabf4201433feeba54fd7ec6145bd5d5654fee5f8e42787a1cf39b3e80e7b1: Status 404 returned error can't find the container with id eceabf4201433feeba54fd7ec6145bd5d5654fee5f8e42787a1cf39b3e80e7b1 Jul 14 21:42:20.306452 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.306431 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 21:42:20.360792 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.360757 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:20.420027 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.419987 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" event={"ID":"1319bf20-cc7d-474e-bd56-89e5dee487b7","Type":"ContainerStarted","Data":"eceabf4201433feeba54fd7ec6145bd5d5654fee5f8e42787a1cf39b3e80e7b1"} Jul 14 21:42:20.464302 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.464258 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:20.561789 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:20.561678 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:22.985743 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:22.985707 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:23.087010 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:23.086973 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:23.437108 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:23.437072 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" event={"ID":"1319bf20-cc7d-474e-bd56-89e5dee487b7","Type":"ContainerStarted","Data":"3e0ae4c3510b84e0c261bb57f2103eee27eaec2ee2e4e67a2fb2caa80d6db537"} Jul 14 21:42:25.447600 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:25.447561 2552 generic.go:356] "Generic (PLEG): container finished" podID="1319bf20-cc7d-474e-bd56-89e5dee487b7" containerID="3e0ae4c3510b84e0c261bb57f2103eee27eaec2ee2e4e67a2fb2caa80d6db537" exitCode=0 Jul 14 21:42:25.448006 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:25.447623 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" event={"ID":"1319bf20-cc7d-474e-bd56-89e5dee487b7","Type":"ContainerDied","Data":"3e0ae4c3510b84e0c261bb57f2103eee27eaec2ee2e4e67a2fb2caa80d6db537"} Jul 14 21:42:26.454479 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:26.454437 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" event={"ID":"1319bf20-cc7d-474e-bd56-89e5dee487b7","Type":"ContainerStarted","Data":"f5bde97453befbdc6d93a4a82a4dbf464e01208d074c0dccde80db2ec0ae2871"} Jul 14 21:42:26.454985 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:26.454682 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:26.485046 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:26.484994 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" podStartSLOduration=3.055478857 podStartE2EDuration="7.484976608s" totalImagesPullingTime="3.300692264s" totalInitContainerRuntime="1.128805487s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:42:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:42:26.48237084 +0000 UTC m=+313.021076523" watchObservedRunningTime="2026-07-14 21:42:26.484976608 +0000 UTC m=+313.023682280" Jul 14 21:42:37.462503 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:37.462461 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c55cf7c4c-zkxmh" Jul 14 21:42:39.150819 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.150782 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:39.512339 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.512248 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65594b5d8c-fbm2h"] Jul 14 21:42:39.515782 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.515756 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65594b5d8c-fbm2h" Jul 14 21:42:39.519460 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.519436 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 14 21:42:39.532498 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.532463 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 14 21:42:39.534404 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.534377 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65594b5d8c-fbm2h"] Jul 14 21:42:39.624745 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.624704 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2769b0-323f-458f-a947-88511ea44ee1-tls-certs\") pod \"kube-auth-proxy-65594b5d8c-fbm2h\" (UID: \"ea2769b0-323f-458f-a947-88511ea44ee1\") " pod="openshift-ingress/kube-auth-proxy-65594b5d8c-fbm2h" Jul 14 21:42:39.624997 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.624818 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s8j5\" (UniqueName: \"kubernetes.io/projected/ea2769b0-323f-458f-a947-88511ea44ee1-kube-api-access-5s8j5\") pod \"kube-auth-proxy-65594b5d8c-fbm2h\" (UID: \"ea2769b0-323f-458f-a947-88511ea44ee1\") " pod="openshift-ingress/kube-auth-proxy-65594b5d8c-fbm2h" Jul 14 21:42:39.624997 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.624877 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea2769b0-323f-458f-a947-88511ea44ee1-tmp\") pod \"kube-auth-proxy-65594b5d8c-fbm2h\" (UID: \"ea2769b0-323f-458f-a947-88511ea44ee1\") " pod="openshift-ingress/kube-auth-proxy-65594b5d8c-fbm2h" Jul 14 21:42:39.725959 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.725920 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea2769b0-323f-458f-a947-88511ea44ee1-tmp\") pod \"kube-auth-proxy-65594b5d8c-fbm2h\" (UID: \"ea2769b0-323f-458f-a947-88511ea44ee1\") " pod="openshift-ingress/kube-auth-proxy-65594b5d8c-fbm2h" Jul 14 21:42:39.726132 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.725972 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2769b0-323f-458f-a947-88511ea44ee1-tls-certs\") pod \"kube-auth-proxy-65594b5d8c-fbm2h\" (UID: \"ea2769b0-323f-458f-a947-88511ea44ee1\") " pod="openshift-ingress/kube-auth-proxy-65594b5d8c-fbm2h" Jul 14 21:42:39.726132 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.726011 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s8j5\" (UniqueName: \"kubernetes.io/projected/ea2769b0-323f-458f-a947-88511ea44ee1-kube-api-access-5s8j5\") pod \"kube-auth-proxy-65594b5d8c-fbm2h\" (UID: \"ea2769b0-323f-458f-a947-88511ea44ee1\") " pod="openshift-ingress/kube-auth-proxy-65594b5d8c-fbm2h" Jul 14 21:42:39.728307 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.728278 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea2769b0-323f-458f-a947-88511ea44ee1-tmp\") pod \"kube-auth-proxy-65594b5d8c-fbm2h\" (UID: \"ea2769b0-323f-458f-a947-88511ea44ee1\") " pod="openshift-ingress/kube-auth-proxy-65594b5d8c-fbm2h" Jul 14 21:42:39.728601 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.728582 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2769b0-323f-458f-a947-88511ea44ee1-tls-certs\") pod \"kube-auth-proxy-65594b5d8c-fbm2h\" (UID: \"ea2769b0-323f-458f-a947-88511ea44ee1\") " pod="openshift-ingress/kube-auth-proxy-65594b5d8c-fbm2h" Jul 14 21:42:39.737605 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.737583 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s8j5\" (UniqueName: \"kubernetes.io/projected/ea2769b0-323f-458f-a947-88511ea44ee1-kube-api-access-5s8j5\") pod \"kube-auth-proxy-65594b5d8c-fbm2h\" (UID: \"ea2769b0-323f-458f-a947-88511ea44ee1\") " pod="openshift-ingress/kube-auth-proxy-65594b5d8c-fbm2h" Jul 14 21:42:39.825776 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.825690 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65594b5d8c-fbm2h" Jul 14 21:42:39.864980 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.864934 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-c45n9"] Jul 14 21:42:39.869232 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.869207 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-c45n9" Jul 14 21:42:39.871851 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.871821 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 14 21:42:39.871979 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.871885 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2xf7v\"" Jul 14 21:42:39.880943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.880914 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-c45n9"] Jul 14 21:42:39.971797 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:39.971770 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65594b5d8c-fbm2h"] Jul 14 21:42:39.973947 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:42:39.973908 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2769b0_323f_458f_a947_88511ea44ee1.slice/crio-e69adb46215a792f8af6f0b8d141b54d9d8fdf7dbeb2dd61c5f9680c8a48e4d5 WatchSource:0}: Error finding container e69adb46215a792f8af6f0b8d141b54d9d8fdf7dbeb2dd61c5f9680c8a48e4d5: Status 404 returned error can't find the container with id e69adb46215a792f8af6f0b8d141b54d9d8fdf7dbeb2dd61c5f9680c8a48e4d5 Jul 14 21:42:40.028442 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.028407 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03b03461-030e-4ae8-9292-f85650d0c18a-cert\") pod \"odh-model-controller-7998c67dc4-c45n9\" (UID: \"03b03461-030e-4ae8-9292-f85650d0c18a\") " pod="opendatahub/odh-model-controller-7998c67dc4-c45n9" Jul 14 21:42:40.028638 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.028548 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whqd4\" (UniqueName: \"kubernetes.io/projected/03b03461-030e-4ae8-9292-f85650d0c18a-kube-api-access-whqd4\") pod \"odh-model-controller-7998c67dc4-c45n9\" (UID: \"03b03461-030e-4ae8-9292-f85650d0c18a\") " pod="opendatahub/odh-model-controller-7998c67dc4-c45n9" Jul 14 21:42:40.129934 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.129894 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03b03461-030e-4ae8-9292-f85650d0c18a-cert\") pod \"odh-model-controller-7998c67dc4-c45n9\" (UID: \"03b03461-030e-4ae8-9292-f85650d0c18a\") " pod="opendatahub/odh-model-controller-7998c67dc4-c45n9" Jul 14 21:42:40.130120 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:42:40.130056 2552 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 14 21:42:40.130167 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:42:40.130125 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03b03461-030e-4ae8-9292-f85650d0c18a-cert podName:03b03461-030e-4ae8-9292-f85650d0c18a nodeName:}" failed. No retries permitted until 2026-07-14 21:42:40.630109128 +0000 UTC m=+327.168814777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/03b03461-030e-4ae8-9292-f85650d0c18a-cert") pod "odh-model-controller-7998c67dc4-c45n9" (UID: "03b03461-030e-4ae8-9292-f85650d0c18a") : secret "odh-model-controller-webhook-cert" not found Jul 14 21:42:40.130247 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.130201 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whqd4\" (UniqueName: \"kubernetes.io/projected/03b03461-030e-4ae8-9292-f85650d0c18a-kube-api-access-whqd4\") pod \"odh-model-controller-7998c67dc4-c45n9\" (UID: \"03b03461-030e-4ae8-9292-f85650d0c18a\") " pod="opendatahub/odh-model-controller-7998c67dc4-c45n9" Jul 14 21:42:40.143726 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.143695 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whqd4\" (UniqueName: \"kubernetes.io/projected/03b03461-030e-4ae8-9292-f85650d0c18a-kube-api-access-whqd4\") pod \"odh-model-controller-7998c67dc4-c45n9\" (UID: \"03b03461-030e-4ae8-9292-f85650d0c18a\") " pod="opendatahub/odh-model-controller-7998c67dc4-c45n9" Jul 14 21:42:40.451717 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.451630 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:40.522683 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.522644 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65594b5d8c-fbm2h" event={"ID":"ea2769b0-323f-458f-a947-88511ea44ee1","Type":"ContainerStarted","Data":"e69adb46215a792f8af6f0b8d141b54d9d8fdf7dbeb2dd61c5f9680c8a48e4d5"} Jul 14 21:42:40.635437 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.635400 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03b03461-030e-4ae8-9292-f85650d0c18a-cert\") pod \"odh-model-controller-7998c67dc4-c45n9\" (UID: \"03b03461-030e-4ae8-9292-f85650d0c18a\") " pod="opendatahub/odh-model-controller-7998c67dc4-c45n9" Jul 14 21:42:40.637852 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.637823 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03b03461-030e-4ae8-9292-f85650d0c18a-cert\") pod \"odh-model-controller-7998c67dc4-c45n9\" (UID: \"03b03461-030e-4ae8-9292-f85650d0c18a\") " pod="opendatahub/odh-model-controller-7998c67dc4-c45n9" Jul 14 21:42:40.785955 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.785584 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-c45n9" Jul 14 21:42:40.806931 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.806896 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-wnrf5"] Jul 14 21:42:40.811301 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.811281 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-wnrf5" Jul 14 21:42:40.814293 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.814257 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 14 21:42:40.823773 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.823627 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-wnrf5"] Jul 14 21:42:40.828371 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.826699 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:40.850637 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.850607 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj"] Jul 14 21:42:40.854751 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.854654 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:40.858910 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.858529 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 14 21:42:40.858910 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.858599 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 14 21:42:40.858910 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.858744 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 21:42:40.858910 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.858766 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 14 21:42:40.859212 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.859145 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fb7wx\"" Jul 14 21:42:40.869826 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.868814 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj"] Jul 14 21:42:40.906780 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.906744 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:40.938482 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.938431 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd94038-ea1c-4009-8e2c-838e8b43f4b5-tls-certs\") pod \"model-serving-api-54d89bd5fc-wnrf5\" (UID: \"6cd94038-ea1c-4009-8e2c-838e8b43f4b5\") " pod="opendatahub/model-serving-api-54d89bd5fc-wnrf5" Jul 14 21:42:40.938482 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.938475 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r69h8\" (UniqueName: \"kubernetes.io/projected/6cd94038-ea1c-4009-8e2c-838e8b43f4b5-kube-api-access-r69h8\") pod \"model-serving-api-54d89bd5fc-wnrf5\" (UID: \"6cd94038-ea1c-4009-8e2c-838e8b43f4b5\") " pod="opendatahub/model-serving-api-54d89bd5fc-wnrf5" Jul 14 21:42:40.958333 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:40.958292 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-c45n9"] Jul 14 21:42:40.961048 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:42:40.961013 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b03461_030e_4ae8_9292_f85650d0c18a.slice/crio-1c9666bdd7d0db778817d727137709cf5cca0fcc26171820f8d738a36d006c8b WatchSource:0}: Error finding container 1c9666bdd7d0db778817d727137709cf5cca0fcc26171820f8d738a36d006c8b: Status 404 returned error can't find the container with id 1c9666bdd7d0db778817d727137709cf5cca0fcc26171820f8d738a36d006c8b Jul 14 21:42:41.039620 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.039531 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/65850aab-f4db-4de8-a298-f281c697271e-local-certs\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.039620 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.039598 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd94038-ea1c-4009-8e2c-838e8b43f4b5-tls-certs\") pod \"model-serving-api-54d89bd5fc-wnrf5\" (UID: \"6cd94038-ea1c-4009-8e2c-838e8b43f4b5\") " pod="opendatahub/model-serving-api-54d89bd5fc-wnrf5" Jul 14 21:42:41.039830 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.039659 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r69h8\" (UniqueName: \"kubernetes.io/projected/6cd94038-ea1c-4009-8e2c-838e8b43f4b5-kube-api-access-r69h8\") pod \"model-serving-api-54d89bd5fc-wnrf5\" (UID: \"6cd94038-ea1c-4009-8e2c-838e8b43f4b5\") " pod="opendatahub/model-serving-api-54d89bd5fc-wnrf5" Jul 14 21:42:41.039830 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.039723 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65850aab-f4db-4de8-a298-f281c697271e-istio-token\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.039830 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.039772 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.039830 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:42:41.039775 2552 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 14 21:42:41.039830 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.039820 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/65850aab-f4db-4de8-a298-f281c697271e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.040088 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:42:41.039887 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cd94038-ea1c-4009-8e2c-838e8b43f4b5-tls-certs podName:6cd94038-ea1c-4009-8e2c-838e8b43f4b5 nodeName:}" failed. No retries permitted until 2026-07-14 21:42:41.539860862 +0000 UTC m=+328.078566526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6cd94038-ea1c-4009-8e2c-838e8b43f4b5-tls-certs") pod "model-serving-api-54d89bd5fc-wnrf5" (UID: "6cd94038-ea1c-4009-8e2c-838e8b43f4b5") : secret "model-serving-api-tls" not found Jul 14 21:42:41.040088 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.039918 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.040088 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.039962 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r85kd\" (UniqueName: \"kubernetes.io/projected/65850aab-f4db-4de8-a298-f281c697271e-kube-api-access-r85kd\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.040088 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.039995 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-cacerts\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.049429 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.049399 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r69h8\" (UniqueName: \"kubernetes.io/projected/6cd94038-ea1c-4009-8e2c-838e8b43f4b5-kube-api-access-r69h8\") pod \"model-serving-api-54d89bd5fc-wnrf5\" (UID: \"6cd94038-ea1c-4009-8e2c-838e8b43f4b5\") " pod="opendatahub/model-serving-api-54d89bd5fc-wnrf5" Jul 14 21:42:41.140564 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.140529 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/65850aab-f4db-4de8-a298-f281c697271e-local-certs\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.140749 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.140592 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65850aab-f4db-4de8-a298-f281c697271e-istio-token\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.140749 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.140616 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.140749 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.140634 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/65850aab-f4db-4de8-a298-f281c697271e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.140749 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.140653 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.140749 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.140679 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r85kd\" (UniqueName: \"kubernetes.io/projected/65850aab-f4db-4de8-a298-f281c697271e-kube-api-access-r85kd\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.140749 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.140699 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-cacerts\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.141382 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.141353 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/65850aab-f4db-4de8-a298-f281c697271e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.143129 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.143097 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/65850aab-f4db-4de8-a298-f281c697271e-local-certs\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.143251 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.143164 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.143372 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.143352 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-cacerts\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.143419 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.143368 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.149808 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.149782 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65850aab-f4db-4de8-a298-f281c697271e-istio-token\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.150117 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.150097 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r85kd\" (UniqueName: \"kubernetes.io/projected/65850aab-f4db-4de8-a298-f281c697271e-kube-api-access-r85kd\") pod \"istiod-openshift-gateway-7574459656-p6zvj\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.171027 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.170999 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:41.207006 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.206968 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:41.313594 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.313556 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj"] Jul 14 21:42:41.316122 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:42:41.316091 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65850aab_f4db_4de8_a298_f281c697271e.slice/crio-e6946fc04e53a9897c099132ba679f9e5e53e1210a0a7db02c6257ad2638af33 WatchSource:0}: Error finding container e6946fc04e53a9897c099132ba679f9e5e53e1210a0a7db02c6257ad2638af33: Status 404 returned error can't find the container with id e6946fc04e53a9897c099132ba679f9e5e53e1210a0a7db02c6257ad2638af33 Jul 14 21:42:41.529765 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.529712 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" event={"ID":"65850aab-f4db-4de8-a298-f281c697271e","Type":"ContainerStarted","Data":"e6946fc04e53a9897c099132ba679f9e5e53e1210a0a7db02c6257ad2638af33"} Jul 14 21:42:41.531153 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.531126 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-c45n9" event={"ID":"03b03461-030e-4ae8-9292-f85650d0c18a","Type":"ContainerStarted","Data":"1c9666bdd7d0db778817d727137709cf5cca0fcc26171820f8d738a36d006c8b"} Jul 14 21:42:41.544721 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.544694 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd94038-ea1c-4009-8e2c-838e8b43f4b5-tls-certs\") pod \"model-serving-api-54d89bd5fc-wnrf5\" (UID: \"6cd94038-ea1c-4009-8e2c-838e8b43f4b5\") " pod="opendatahub/model-serving-api-54d89bd5fc-wnrf5" Jul 14 21:42:41.547484 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.547460 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd94038-ea1c-4009-8e2c-838e8b43f4b5-tls-certs\") pod \"model-serving-api-54d89bd5fc-wnrf5\" (UID: \"6cd94038-ea1c-4009-8e2c-838e8b43f4b5\") " pod="opendatahub/model-serving-api-54d89bd5fc-wnrf5" Jul 14 21:42:41.749825 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.749567 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-wnrf5" Jul 14 21:42:41.950313 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.950179 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-wnrf5"] Jul 14 21:42:41.954585 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:42:41.954479 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd94038_ea1c_4009_8e2c_838e8b43f4b5.slice/crio-d75054fb1ed7ffed0ecda57a0be228b89c08953224fe12d55a90af129c139658 WatchSource:0}: Error finding container d75054fb1ed7ffed0ecda57a0be228b89c08953224fe12d55a90af129c139658: Status 404 returned error can't find the container with id d75054fb1ed7ffed0ecda57a0be228b89c08953224fe12d55a90af129c139658 Jul 14 21:42:41.959309 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:41.959236 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:42.542949 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:42.542899 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-wnrf5" event={"ID":"6cd94038-ea1c-4009-8e2c-838e8b43f4b5","Type":"ContainerStarted","Data":"d75054fb1ed7ffed0ecda57a0be228b89c08953224fe12d55a90af129c139658"} Jul 14 21:42:45.833776 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:45.833722 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 21:42:45.834121 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:45.833812 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 21:42:46.567254 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:46.567213 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-c45n9" event={"ID":"03b03461-030e-4ae8-9292-f85650d0c18a","Type":"ContainerStarted","Data":"d614810d11485f9e4d0b2b390140db413ab5f06e04097f630521057be5a6fa75"} Jul 14 21:42:46.567461 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:46.567295 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-c45n9" Jul 14 21:42:46.569221 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:46.569177 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-wnrf5" event={"ID":"6cd94038-ea1c-4009-8e2c-838e8b43f4b5","Type":"ContainerStarted","Data":"d1cc753f171037e5c8b7b8c8acc781dc1ecfa5295ae9002ca676454a3466f464"} Jul 14 21:42:46.569386 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:46.569245 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-wnrf5" Jul 14 21:42:46.571068 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:46.571036 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65594b5d8c-fbm2h" event={"ID":"ea2769b0-323f-458f-a947-88511ea44ee1","Type":"ContainerStarted","Data":"6c71f9365c5e804052b21058c5438e67436d1a2970292b40972f2d34f848e811"} Jul 14 21:42:46.573077 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:46.573050 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" event={"ID":"65850aab-f4db-4de8-a298-f281c697271e","Type":"ContainerStarted","Data":"6742a9478cc8733abe5f9d3a9cf2e8d6a465465cee04cd97297b9c5a71ff1af2"} Jul 14 21:42:46.573222 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:46.573173 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:46.574996 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:46.574971 2552 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-p6zvj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 21:42:46.575085 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:46.575017 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" podUID="65850aab-f4db-4de8-a298-f281c697271e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 21:42:46.587609 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:46.587565 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-c45n9" podStartSLOduration=2.707606333 podStartE2EDuration="7.587551502s" totalImagesPullingTime="4.879945169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:42:46.585508126 +0000 UTC m=+333.124213798" watchObservedRunningTime="2026-07-14 21:42:46.587551502 +0000 UTC m=+333.126257227" Jul 14 21:42:46.603784 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:46.603724 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-wnrf5" podStartSLOduration=2.722462519 podStartE2EDuration="6.603705486s" totalImagesPullingTime="3.881242967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:42:46.602781946 +0000 UTC m=+333.141487618" watchObservedRunningTime="2026-07-14 21:42:46.603705486 +0000 UTC m=+333.142411159" Jul 14 21:42:46.620693 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:46.620632 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65594b5d8c-fbm2h" podStartSLOduration=1.770368864 podStartE2EDuration="7.620617467s" totalImagesPullingTime="5.850248603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:42:46.618725064 +0000 UTC m=+333.157430737" watchObservedRunningTime="2026-07-14 21:42:46.620617467 +0000 UTC m=+333.159323177" Jul 14 21:42:46.640249 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:46.640178 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" podStartSLOduration=2.1254689940000002 podStartE2EDuration="6.640162105s" totalImagesPullingTime="4.514693111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:42:46.638125796 +0000 UTC m=+333.176831493" watchObservedRunningTime="2026-07-14 21:42:46.640162105 +0000 UTC m=+333.178867776" Jul 14 21:42:47.237931 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:47.237893 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:47.360247 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:47.360211 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:47.578495 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:47.578413 2552 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-p6zvj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 21:42:47.578495 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:47.578466 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" podUID="65850aab-f4db-4de8-a298-f281c697271e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 21:42:47.681792 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:47.681748 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:47.780132 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:47.780088 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:47.879763 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:47.879727 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:49.310995 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.310957 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6"] Jul 14 21:42:49.314058 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.314030 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" Jul 14 21:42:49.317308 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.317271 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 14 21:42:49.317455 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.317332 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 14 21:42:49.317455 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.317387 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 14 21:42:49.318337 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.318316 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 14 21:42:49.318448 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.318415 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 14 21:42:49.318548 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.318487 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rlqf6\"" Jul 14 21:42:49.327341 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.327319 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6"] Jul 14 21:42:49.413448 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.413412 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/025a34dc-2764-4ca3-a424-53b872ced886-metrics-cert\") pod \"lws-controller-manager-65d6b5d88-8q7q6\" (UID: \"025a34dc-2764-4ca3-a424-53b872ced886\") " pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" Jul 14 21:42:49.413448 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.413462 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/025a34dc-2764-4ca3-a424-53b872ced886-cert\") pod \"lws-controller-manager-65d6b5d88-8q7q6\" (UID: \"025a34dc-2764-4ca3-a424-53b872ced886\") " pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" Jul 14 21:42:49.413672 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.413488 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wlqz\" (UniqueName: \"kubernetes.io/projected/025a34dc-2764-4ca3-a424-53b872ced886-kube-api-access-7wlqz\") pod \"lws-controller-manager-65d6b5d88-8q7q6\" (UID: \"025a34dc-2764-4ca3-a424-53b872ced886\") " pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" Jul 14 21:42:49.413672 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.413509 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/025a34dc-2764-4ca3-a424-53b872ced886-manager-config\") pod \"lws-controller-manager-65d6b5d88-8q7q6\" (UID: \"025a34dc-2764-4ca3-a424-53b872ced886\") " pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" Jul 14 21:42:49.514000 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.513962 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/025a34dc-2764-4ca3-a424-53b872ced886-cert\") pod \"lws-controller-manager-65d6b5d88-8q7q6\" (UID: \"025a34dc-2764-4ca3-a424-53b872ced886\") " pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" Jul 14 21:42:49.514176 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.514010 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wlqz\" (UniqueName: \"kubernetes.io/projected/025a34dc-2764-4ca3-a424-53b872ced886-kube-api-access-7wlqz\") pod \"lws-controller-manager-65d6b5d88-8q7q6\" (UID: \"025a34dc-2764-4ca3-a424-53b872ced886\") " pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" Jul 14 21:42:49.514176 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.514138 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/025a34dc-2764-4ca3-a424-53b872ced886-manager-config\") pod \"lws-controller-manager-65d6b5d88-8q7q6\" (UID: \"025a34dc-2764-4ca3-a424-53b872ced886\") " pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" Jul 14 21:42:49.514324 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.514265 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/025a34dc-2764-4ca3-a424-53b872ced886-metrics-cert\") pod \"lws-controller-manager-65d6b5d88-8q7q6\" (UID: \"025a34dc-2764-4ca3-a424-53b872ced886\") " pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" Jul 14 21:42:49.514953 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.514892 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/025a34dc-2764-4ca3-a424-53b872ced886-manager-config\") pod \"lws-controller-manager-65d6b5d88-8q7q6\" (UID: \"025a34dc-2764-4ca3-a424-53b872ced886\") " pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" Jul 14 21:42:49.516649 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.516626 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/025a34dc-2764-4ca3-a424-53b872ced886-metrics-cert\") pod \"lws-controller-manager-65d6b5d88-8q7q6\" (UID: \"025a34dc-2764-4ca3-a424-53b872ced886\") " pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" Jul 14 21:42:49.516769 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.516753 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/025a34dc-2764-4ca3-a424-53b872ced886-cert\") pod \"lws-controller-manager-65d6b5d88-8q7q6\" (UID: \"025a34dc-2764-4ca3-a424-53b872ced886\") " pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" Jul 14 21:42:49.532268 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.532245 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wlqz\" (UniqueName: \"kubernetes.io/projected/025a34dc-2764-4ca3-a424-53b872ced886-kube-api-access-7wlqz\") pod \"lws-controller-manager-65d6b5d88-8q7q6\" (UID: \"025a34dc-2764-4ca3-a424-53b872ced886\") " pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" Jul 14 21:42:49.624130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.624101 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" Jul 14 21:42:49.760296 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:49.760270 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6"] Jul 14 21:42:49.762119 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:42:49.762094 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025a34dc_2764_4ca3_a424_53b872ced886.slice/crio-13563a0f8dfe5f1517ed560e5e6308b05f1abe74f7f7bdbba81fac938b1c6ffe WatchSource:0}: Error finding container 13563a0f8dfe5f1517ed560e5e6308b05f1abe74f7f7bdbba81fac938b1c6ffe: Status 404 returned error can't find the container with id 13563a0f8dfe5f1517ed560e5e6308b05f1abe74f7f7bdbba81fac938b1c6ffe Jul 14 21:42:50.579483 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:50.579396 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:50.600887 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:50.600842 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" event={"ID":"025a34dc-2764-4ca3-a424-53b872ced886","Type":"ContainerStarted","Data":"13563a0f8dfe5f1517ed560e5e6308b05f1abe74f7f7bdbba81fac938b1c6ffe"} Jul 14 21:42:50.612076 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:50.611990 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:50.703147 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:50.703102 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:50.764081 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:50.764044 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:50.870687 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:50.870649 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:51.601044 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:51.601007 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:51.606737 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:51.606707 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" event={"ID":"025a34dc-2764-4ca3-a424-53b872ced886","Type":"ContainerStarted","Data":"48e789d5e84198b7568e00bd0253eecaed375af04854a9c7c61bd55b3159aa38"} Jul 14 21:42:51.606889 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:51.606802 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" Jul 14 21:42:51.627520 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:51.627474 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" podStartSLOduration=1.066502803 podStartE2EDuration="2.627456183s" totalImagesPullingTime="1.56095338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:42:51.627325473 +0000 UTC m=+338.166031146" watchObservedRunningTime="2026-07-14 21:42:51.627456183 +0000 UTC m=+338.166161854" Jul 14 21:42:51.927170 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:51.927082 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:52.021157 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.021124 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:52.115739 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.115710 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj"] Jul 14 21:42:52.115908 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.115891 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" podUID="65850aab-f4db-4de8-a298-f281c697271e" containerName="discovery" containerID="cri-o://6742a9478cc8733abe5f9d3a9cf2e8d6a465465cee04cd97297b9c5a71ff1af2" gracePeriod=30 Jul 14 21:42:52.127500 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.127460 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:52.360822 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.360798 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:52.446344 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.446305 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-cacerts\") pod \"65850aab-f4db-4de8-a298-f281c697271e\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " Jul 14 21:42:52.446521 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.446362 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/65850aab-f4db-4de8-a298-f281c697271e-istio-csr-ca-configmap\") pod \"65850aab-f4db-4de8-a298-f281c697271e\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " Jul 14 21:42:52.446521 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.446394 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r85kd\" (UniqueName: \"kubernetes.io/projected/65850aab-f4db-4de8-a298-f281c697271e-kube-api-access-r85kd\") pod \"65850aab-f4db-4de8-a298-f281c697271e\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " Jul 14 21:42:52.446521 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.446439 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/65850aab-f4db-4de8-a298-f281c697271e-local-certs\") pod \"65850aab-f4db-4de8-a298-f281c697271e\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " Jul 14 21:42:52.446521 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.446470 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65850aab-f4db-4de8-a298-f281c697271e-istio-token\") pod \"65850aab-f4db-4de8-a298-f281c697271e\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " Jul 14 21:42:52.446702 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.446543 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-istio-csr-dns-cert\") pod \"65850aab-f4db-4de8-a298-f281c697271e\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " Jul 14 21:42:52.446702 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.446573 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-istio-kubeconfig\") pod \"65850aab-f4db-4de8-a298-f281c697271e\" (UID: \"65850aab-f4db-4de8-a298-f281c697271e\") " Jul 14 21:42:52.446849 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.446823 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65850aab-f4db-4de8-a298-f281c697271e-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "65850aab-f4db-4de8-a298-f281c697271e" (UID: "65850aab-f4db-4de8-a298-f281c697271e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:42:52.449000 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.448964 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "65850aab-f4db-4de8-a298-f281c697271e" (UID: "65850aab-f4db-4de8-a298-f281c697271e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:42:52.449140 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.449025 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "65850aab-f4db-4de8-a298-f281c697271e" (UID: "65850aab-f4db-4de8-a298-f281c697271e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:42:52.449140 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.449074 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-cacerts" (OuterVolumeSpecName: "cacerts") pod "65850aab-f4db-4de8-a298-f281c697271e" (UID: "65850aab-f4db-4de8-a298-f281c697271e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:42:52.449140 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.449095 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65850aab-f4db-4de8-a298-f281c697271e-local-certs" (OuterVolumeSpecName: "local-certs") pod "65850aab-f4db-4de8-a298-f281c697271e" (UID: "65850aab-f4db-4de8-a298-f281c697271e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:42:52.449296 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.449148 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65850aab-f4db-4de8-a298-f281c697271e-kube-api-access-r85kd" (OuterVolumeSpecName: "kube-api-access-r85kd") pod "65850aab-f4db-4de8-a298-f281c697271e" (UID: "65850aab-f4db-4de8-a298-f281c697271e"). InnerVolumeSpecName "kube-api-access-r85kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:42:52.449333 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.449292 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65850aab-f4db-4de8-a298-f281c697271e-istio-token" (OuterVolumeSpecName: "istio-token") pod "65850aab-f4db-4de8-a298-f281c697271e" (UID: "65850aab-f4db-4de8-a298-f281c697271e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:42:52.548002 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.547907 2552 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-istio-csr-dns-cert\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:42:52.548002 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.547939 2552 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-istio-kubeconfig\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:42:52.548002 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.547950 2552 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/65850aab-f4db-4de8-a298-f281c697271e-cacerts\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:42:52.548002 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.547964 2552 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/65850aab-f4db-4de8-a298-f281c697271e-istio-csr-ca-configmap\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:42:52.548002 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.547974 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r85kd\" (UniqueName: \"kubernetes.io/projected/65850aab-f4db-4de8-a298-f281c697271e-kube-api-access-r85kd\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:42:52.548002 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.547983 2552 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/65850aab-f4db-4de8-a298-f281c697271e-local-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:42:52.548002 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.547992 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65850aab-f4db-4de8-a298-f281c697271e-istio-token\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:42:52.616383 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.616345 2552 generic.go:356] "Generic (PLEG): container finished" podID="65850aab-f4db-4de8-a298-f281c697271e" containerID="6742a9478cc8733abe5f9d3a9cf2e8d6a465465cee04cd97297b9c5a71ff1af2" exitCode=0 Jul 14 21:42:52.616867 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.616398 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" Jul 14 21:42:52.616867 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.616431 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" event={"ID":"65850aab-f4db-4de8-a298-f281c697271e","Type":"ContainerDied","Data":"6742a9478cc8733abe5f9d3a9cf2e8d6a465465cee04cd97297b9c5a71ff1af2"} Jul 14 21:42:52.616867 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.616482 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj" event={"ID":"65850aab-f4db-4de8-a298-f281c697271e","Type":"ContainerDied","Data":"e6946fc04e53a9897c099132ba679f9e5e53e1210a0a7db02c6257ad2638af33"} Jul 14 21:42:52.616867 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.616502 2552 scope.go:117] "RemoveContainer" containerID="6742a9478cc8733abe5f9d3a9cf2e8d6a465465cee04cd97297b9c5a71ff1af2" Jul 14 21:42:52.625994 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.625974 2552 scope.go:117] "RemoveContainer" containerID="6742a9478cc8733abe5f9d3a9cf2e8d6a465465cee04cd97297b9c5a71ff1af2" Jul 14 21:42:52.626335 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:42:52.626313 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6742a9478cc8733abe5f9d3a9cf2e8d6a465465cee04cd97297b9c5a71ff1af2\": container with ID starting with 6742a9478cc8733abe5f9d3a9cf2e8d6a465465cee04cd97297b9c5a71ff1af2 not found: ID does not exist" containerID="6742a9478cc8733abe5f9d3a9cf2e8d6a465465cee04cd97297b9c5a71ff1af2" Jul 14 21:42:52.626401 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.626345 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6742a9478cc8733abe5f9d3a9cf2e8d6a465465cee04cd97297b9c5a71ff1af2"} err="failed to get container status \"6742a9478cc8733abe5f9d3a9cf2e8d6a465465cee04cd97297b9c5a71ff1af2\": rpc error: code = NotFound desc = could not find container \"6742a9478cc8733abe5f9d3a9cf2e8d6a465465cee04cd97297b9c5a71ff1af2\": container with ID starting with 6742a9478cc8733abe5f9d3a9cf2e8d6a465465cee04cd97297b9c5a71ff1af2 not found: ID does not exist" Jul 14 21:42:52.641920 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.641887 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj"] Jul 14 21:42:52.650324 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:52.650296 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-p6zvj"] Jul 14 21:42:53.992368 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:53.992332 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65850aab-f4db-4de8-a298-f281c697271e" path="/var/lib/kubelet/pods/65850aab-f4db-4de8-a298-f281c697271e/volumes" Jul 14 21:42:54.526647 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:54.526604 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:42:57.580393 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:57.580354 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-c45n9" Jul 14 21:42:57.582274 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:42:57.582253 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-wnrf5" Jul 14 21:43:02.619121 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:02.619083 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-65d6b5d88-8q7q6" Jul 14 21:43:29.110798 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:29.110757 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:43:34.671838 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:34.671802 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:43:34.726001 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:34.725968 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:43:36.611293 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:36.611250 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:43:40.715596 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:40.715551 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:43:40.811828 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:40.811792 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:43:40.964477 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:40.964441 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:43:55.860680 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:55.860638 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9"] Jul 14 21:43:55.861059 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:55.860965 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65850aab-f4db-4de8-a298-f281c697271e" containerName="discovery" Jul 14 21:43:55.861059 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:55.860976 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="65850aab-f4db-4de8-a298-f281c697271e" containerName="discovery" Jul 14 21:43:55.861059 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:55.861054 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="65850aab-f4db-4de8-a298-f281c697271e" containerName="discovery" Jul 14 21:43:55.866513 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:55.866479 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" Jul 14 21:43:55.873726 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:55.873694 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 21:43:55.873969 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:55.873722 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 21:43:55.874816 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:55.874789 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mgtcm\"" Jul 14 21:43:55.877436 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:55.877414 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9"] Jul 14 21:43:55.984752 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:55.984706 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z8vw\" (UniqueName: \"kubernetes.io/projected/3446180e-7b30-45e7-a5e9-118a1384d1e7-kube-api-access-7z8vw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9\" (UID: \"3446180e-7b30-45e7-a5e9-118a1384d1e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" Jul 14 21:43:55.984929 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:55.984785 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3446180e-7b30-45e7-a5e9-118a1384d1e7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9\" (UID: \"3446180e-7b30-45e7-a5e9-118a1384d1e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" Jul 14 21:43:55.984929 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:55.984823 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3446180e-7b30-45e7-a5e9-118a1384d1e7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9\" (UID: \"3446180e-7b30-45e7-a5e9-118a1384d1e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" Jul 14 21:43:56.085747 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.085714 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3446180e-7b30-45e7-a5e9-118a1384d1e7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9\" (UID: \"3446180e-7b30-45e7-a5e9-118a1384d1e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" Jul 14 21:43:56.085894 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.085764 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3446180e-7b30-45e7-a5e9-118a1384d1e7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9\" (UID: \"3446180e-7b30-45e7-a5e9-118a1384d1e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" Jul 14 21:43:56.085894 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.085829 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z8vw\" (UniqueName: \"kubernetes.io/projected/3446180e-7b30-45e7-a5e9-118a1384d1e7-kube-api-access-7z8vw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9\" (UID: \"3446180e-7b30-45e7-a5e9-118a1384d1e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" Jul 14 21:43:56.086112 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.086094 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3446180e-7b30-45e7-a5e9-118a1384d1e7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9\" (UID: \"3446180e-7b30-45e7-a5e9-118a1384d1e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" Jul 14 21:43:56.086154 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.086128 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3446180e-7b30-45e7-a5e9-118a1384d1e7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9\" (UID: \"3446180e-7b30-45e7-a5e9-118a1384d1e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" Jul 14 21:43:56.096425 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.096401 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z8vw\" (UniqueName: \"kubernetes.io/projected/3446180e-7b30-45e7-a5e9-118a1384d1e7-kube-api-access-7z8vw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9\" (UID: \"3446180e-7b30-45e7-a5e9-118a1384d1e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" Jul 14 21:43:56.176398 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.176310 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" Jul 14 21:43:56.320644 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.320616 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9"] Jul 14 21:43:56.322670 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:43:56.322643 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3446180e_7b30_45e7_a5e9_118a1384d1e7.slice/crio-0f57c453c8e7ebbfaa12124f6af41eba0953601e2fb296532eecca8584fc80aa WatchSource:0}: Error finding container 0f57c453c8e7ebbfaa12124f6af41eba0953601e2fb296532eecca8584fc80aa: Status 404 returned error can't find the container with id 0f57c453c8e7ebbfaa12124f6af41eba0953601e2fb296532eecca8584fc80aa Jul 14 21:43:56.463701 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.463665 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v"] Jul 14 21:43:56.467105 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.467087 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" Jul 14 21:43:56.477827 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.477800 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v"] Jul 14 21:43:56.590537 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.590486 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d93ce281-4e29-4317-a841-8f530d9dc34d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v\" (UID: \"d93ce281-4e29-4317-a841-8f530d9dc34d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" Jul 14 21:43:56.590537 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.590540 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhkvr\" (UniqueName: \"kubernetes.io/projected/d93ce281-4e29-4317-a841-8f530d9dc34d-kube-api-access-rhkvr\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v\" (UID: \"d93ce281-4e29-4317-a841-8f530d9dc34d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" Jul 14 21:43:56.590813 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.590666 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d93ce281-4e29-4317-a841-8f530d9dc34d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v\" (UID: \"d93ce281-4e29-4317-a841-8f530d9dc34d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" Jul 14 21:43:56.691928 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.691827 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d93ce281-4e29-4317-a841-8f530d9dc34d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v\" (UID: \"d93ce281-4e29-4317-a841-8f530d9dc34d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" Jul 14 21:43:56.691928 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.691897 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d93ce281-4e29-4317-a841-8f530d9dc34d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v\" (UID: \"d93ce281-4e29-4317-a841-8f530d9dc34d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" Jul 14 21:43:56.691928 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.691914 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhkvr\" (UniqueName: \"kubernetes.io/projected/d93ce281-4e29-4317-a841-8f530d9dc34d-kube-api-access-rhkvr\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v\" (UID: \"d93ce281-4e29-4317-a841-8f530d9dc34d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" Jul 14 21:43:56.692368 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.692346 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d93ce281-4e29-4317-a841-8f530d9dc34d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v\" (UID: \"d93ce281-4e29-4317-a841-8f530d9dc34d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" Jul 14 21:43:56.692368 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.692357 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d93ce281-4e29-4317-a841-8f530d9dc34d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v\" (UID: \"d93ce281-4e29-4317-a841-8f530d9dc34d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" Jul 14 21:43:56.702769 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.702737 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhkvr\" (UniqueName: \"kubernetes.io/projected/d93ce281-4e29-4317-a841-8f530d9dc34d-kube-api-access-rhkvr\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v\" (UID: \"d93ce281-4e29-4317-a841-8f530d9dc34d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" Jul 14 21:43:56.776793 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.776754 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" Jul 14 21:43:56.915875 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.915845 2552 generic.go:356] "Generic (PLEG): container finished" podID="3446180e-7b30-45e7-a5e9-118a1384d1e7" containerID="d46cfed7ddaaeec3ab352d00b1f9511eced9ec8209a7fdec7ded29ef435d7780" exitCode=0 Jul 14 21:43:56.916166 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.915924 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" event={"ID":"3446180e-7b30-45e7-a5e9-118a1384d1e7","Type":"ContainerDied","Data":"d46cfed7ddaaeec3ab352d00b1f9511eced9ec8209a7fdec7ded29ef435d7780"} Jul 14 21:43:56.916166 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.915983 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" event={"ID":"3446180e-7b30-45e7-a5e9-118a1384d1e7","Type":"ContainerStarted","Data":"0f57c453c8e7ebbfaa12124f6af41eba0953601e2fb296532eecca8584fc80aa"} Jul 14 21:43:56.916272 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:56.916175 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v"] Jul 14 21:43:56.918085 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:43:56.918062 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93ce281_4e29_4317_a841_8f530d9dc34d.slice/crio-b397e43b95dd9ac953ee004674ec2b061d9dc6916db2555705593e5c26fe9ca5 WatchSource:0}: Error finding container b397e43b95dd9ac953ee004674ec2b061d9dc6916db2555705593e5c26fe9ca5: Status 404 returned error can't find the container with id b397e43b95dd9ac953ee004674ec2b061d9dc6916db2555705593e5c26fe9ca5 Jul 14 21:43:57.060405 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.060372 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g"] Jul 14 21:43:57.063829 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.063809 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" Jul 14 21:43:57.073334 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.073302 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g"] Jul 14 21:43:57.095892 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.095861 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4msh2\" (UniqueName: \"kubernetes.io/projected/c63ce66e-682c-4e21-ac5b-b7d748d66190-kube-api-access-4msh2\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g\" (UID: \"c63ce66e-682c-4e21-ac5b-b7d748d66190\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" Jul 14 21:43:57.096059 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.095920 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c63ce66e-682c-4e21-ac5b-b7d748d66190-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g\" (UID: \"c63ce66e-682c-4e21-ac5b-b7d748d66190\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" Jul 14 21:43:57.096059 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.095955 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c63ce66e-682c-4e21-ac5b-b7d748d66190-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g\" (UID: \"c63ce66e-682c-4e21-ac5b-b7d748d66190\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" Jul 14 21:43:57.196985 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.196942 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4msh2\" (UniqueName: \"kubernetes.io/projected/c63ce66e-682c-4e21-ac5b-b7d748d66190-kube-api-access-4msh2\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g\" (UID: \"c63ce66e-682c-4e21-ac5b-b7d748d66190\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" Jul 14 21:43:57.197216 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.197008 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c63ce66e-682c-4e21-ac5b-b7d748d66190-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g\" (UID: \"c63ce66e-682c-4e21-ac5b-b7d748d66190\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" Jul 14 21:43:57.197216 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.197046 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c63ce66e-682c-4e21-ac5b-b7d748d66190-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g\" (UID: \"c63ce66e-682c-4e21-ac5b-b7d748d66190\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" Jul 14 21:43:57.197475 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.197455 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c63ce66e-682c-4e21-ac5b-b7d748d66190-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g\" (UID: \"c63ce66e-682c-4e21-ac5b-b7d748d66190\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" Jul 14 21:43:57.197539 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.197495 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c63ce66e-682c-4e21-ac5b-b7d748d66190-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g\" (UID: \"c63ce66e-682c-4e21-ac5b-b7d748d66190\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" Jul 14 21:43:57.207000 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.206935 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4msh2\" (UniqueName: \"kubernetes.io/projected/c63ce66e-682c-4e21-ac5b-b7d748d66190-kube-api-access-4msh2\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g\" (UID: \"c63ce66e-682c-4e21-ac5b-b7d748d66190\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" Jul 14 21:43:57.375461 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.375422 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" Jul 14 21:43:57.515285 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.515249 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g"] Jul 14 21:43:57.518625 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:43:57.518596 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63ce66e_682c_4e21_ac5b_b7d748d66190.slice/crio-924a1f33ffaa3f4f3dafcb571b07848a9ae90e5e9df8aded39cc86f3d3e9a4fa WatchSource:0}: Error finding container 924a1f33ffaa3f4f3dafcb571b07848a9ae90e5e9df8aded39cc86f3d3e9a4fa: Status 404 returned error can't find the container with id 924a1f33ffaa3f4f3dafcb571b07848a9ae90e5e9df8aded39cc86f3d3e9a4fa Jul 14 21:43:57.660826 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.660792 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx"] Jul 14 21:43:57.664293 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.664269 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" Jul 14 21:43:57.673716 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.673687 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx"] Jul 14 21:43:57.701521 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.701489 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00780a1c-747c-41fa-a237-16fd726b14d3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx\" (UID: \"00780a1c-747c-41fa-a237-16fd726b14d3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" Jul 14 21:43:57.701521 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.701528 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbrp7\" (UniqueName: \"kubernetes.io/projected/00780a1c-747c-41fa-a237-16fd726b14d3-kube-api-access-qbrp7\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx\" (UID: \"00780a1c-747c-41fa-a237-16fd726b14d3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" Jul 14 21:43:57.701736 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.701549 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00780a1c-747c-41fa-a237-16fd726b14d3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx\" (UID: \"00780a1c-747c-41fa-a237-16fd726b14d3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" Jul 14 21:43:57.802878 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.802848 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00780a1c-747c-41fa-a237-16fd726b14d3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx\" (UID: \"00780a1c-747c-41fa-a237-16fd726b14d3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" Jul 14 21:43:57.803020 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.802904 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbrp7\" (UniqueName: \"kubernetes.io/projected/00780a1c-747c-41fa-a237-16fd726b14d3-kube-api-access-qbrp7\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx\" (UID: \"00780a1c-747c-41fa-a237-16fd726b14d3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" Jul 14 21:43:57.803020 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.802944 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00780a1c-747c-41fa-a237-16fd726b14d3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx\" (UID: \"00780a1c-747c-41fa-a237-16fd726b14d3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" Jul 14 21:43:57.803322 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.803297 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00780a1c-747c-41fa-a237-16fd726b14d3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx\" (UID: \"00780a1c-747c-41fa-a237-16fd726b14d3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" Jul 14 21:43:57.803322 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.803315 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00780a1c-747c-41fa-a237-16fd726b14d3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx\" (UID: \"00780a1c-747c-41fa-a237-16fd726b14d3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" Jul 14 21:43:57.812398 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.812378 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbrp7\" (UniqueName: \"kubernetes.io/projected/00780a1c-747c-41fa-a237-16fd726b14d3-kube-api-access-qbrp7\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx\" (UID: \"00780a1c-747c-41fa-a237-16fd726b14d3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" Jul 14 21:43:57.923177 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.923140 2552 generic.go:356] "Generic (PLEG): container finished" podID="3446180e-7b30-45e7-a5e9-118a1384d1e7" containerID="38f0bcac30b61e6031af091da8c4be9677cdb5f61814fdf44add416e10413946" exitCode=0 Jul 14 21:43:57.923619 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.923225 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" event={"ID":"3446180e-7b30-45e7-a5e9-118a1384d1e7","Type":"ContainerDied","Data":"38f0bcac30b61e6031af091da8c4be9677cdb5f61814fdf44add416e10413946"} Jul 14 21:43:57.924944 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.924908 2552 generic.go:356] "Generic (PLEG): container finished" podID="c63ce66e-682c-4e21-ac5b-b7d748d66190" containerID="b2bb21587a0fa6154b70b94bd940c0fa756b1d4090aff6476690d68165ea8262" exitCode=0 Jul 14 21:43:57.925007 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.924984 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" event={"ID":"c63ce66e-682c-4e21-ac5b-b7d748d66190","Type":"ContainerDied","Data":"b2bb21587a0fa6154b70b94bd940c0fa756b1d4090aff6476690d68165ea8262"} Jul 14 21:43:57.925048 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.925034 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" event={"ID":"c63ce66e-682c-4e21-ac5b-b7d748d66190","Type":"ContainerStarted","Data":"924a1f33ffaa3f4f3dafcb571b07848a9ae90e5e9df8aded39cc86f3d3e9a4fa"} Jul 14 21:43:57.928871 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.928845 2552 generic.go:356] "Generic (PLEG): container finished" podID="d93ce281-4e29-4317-a841-8f530d9dc34d" containerID="b9fefde83e121f75fe0cf303f75f9a98f137ca8901c4e69e7253396cbc7b71dd" exitCode=0 Jul 14 21:43:57.928993 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.928915 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" event={"ID":"d93ce281-4e29-4317-a841-8f530d9dc34d","Type":"ContainerDied","Data":"b9fefde83e121f75fe0cf303f75f9a98f137ca8901c4e69e7253396cbc7b71dd"} Jul 14 21:43:57.928993 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.928948 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" event={"ID":"d93ce281-4e29-4317-a841-8f530d9dc34d","Type":"ContainerStarted","Data":"b397e43b95dd9ac953ee004674ec2b061d9dc6916db2555705593e5c26fe9ca5"} Jul 14 21:43:57.984520 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:57.984492 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" Jul 14 21:43:58.166080 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:58.166049 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx"] Jul 14 21:43:58.938743 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:58.938677 2552 generic.go:356] "Generic (PLEG): container finished" podID="3446180e-7b30-45e7-a5e9-118a1384d1e7" containerID="a9541f567b10560f29fc19b5efba44f6fd1d7ed3fe21ff0424f9321851f6c257" exitCode=0 Jul 14 21:43:58.939134 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:58.938761 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" event={"ID":"3446180e-7b30-45e7-a5e9-118a1384d1e7","Type":"ContainerDied","Data":"a9541f567b10560f29fc19b5efba44f6fd1d7ed3fe21ff0424f9321851f6c257"} Jul 14 21:43:58.940310 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:58.940178 2552 generic.go:356] "Generic (PLEG): container finished" podID="00780a1c-747c-41fa-a237-16fd726b14d3" containerID="0c69d3e3eef339461d7a43a7212073860dce5bbc19855a9221892684143ab0ea" exitCode=0 Jul 14 21:43:58.940310 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:58.940220 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" event={"ID":"00780a1c-747c-41fa-a237-16fd726b14d3","Type":"ContainerDied","Data":"0c69d3e3eef339461d7a43a7212073860dce5bbc19855a9221892684143ab0ea"} Jul 14 21:43:58.940310 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:58.940267 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" event={"ID":"00780a1c-747c-41fa-a237-16fd726b14d3","Type":"ContainerStarted","Data":"a87a473a7f2474ceecdde3e0ee7d902ea1b743d4fa6b73a84ccce30d175afa80"} Jul 14 21:43:58.942232 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:58.942184 2552 generic.go:356] "Generic (PLEG): container finished" podID="d93ce281-4e29-4317-a841-8f530d9dc34d" containerID="30eda6cbcc07eeb6e109dd5a7289c95a45b3348395a498a47a68018b8d96d002" exitCode=0 Jul 14 21:43:58.942346 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:58.942226 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" event={"ID":"d93ce281-4e29-4317-a841-8f530d9dc34d","Type":"ContainerDied","Data":"30eda6cbcc07eeb6e109dd5a7289c95a45b3348395a498a47a68018b8d96d002"} Jul 14 21:43:59.949534 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:59.949434 2552 generic.go:356] "Generic (PLEG): container finished" podID="00780a1c-747c-41fa-a237-16fd726b14d3" containerID="7385d77c298b402880aeb54df16d87e06c0f0356708b793decd313f0a2e8e3aa" exitCode=0 Jul 14 21:43:59.950074 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:59.949526 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" event={"ID":"00780a1c-747c-41fa-a237-16fd726b14d3","Type":"ContainerDied","Data":"7385d77c298b402880aeb54df16d87e06c0f0356708b793decd313f0a2e8e3aa"} Jul 14 21:43:59.951570 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:59.951547 2552 generic.go:356] "Generic (PLEG): container finished" podID="c63ce66e-682c-4e21-ac5b-b7d748d66190" containerID="8bb7f6828d4ca5f49092afeaea472990c3f041c216fdca01f58569d2851e7ac7" exitCode=0 Jul 14 21:43:59.951672 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:59.951615 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" event={"ID":"c63ce66e-682c-4e21-ac5b-b7d748d66190","Type":"ContainerDied","Data":"8bb7f6828d4ca5f49092afeaea472990c3f041c216fdca01f58569d2851e7ac7"} Jul 14 21:43:59.953712 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:59.953692 2552 generic.go:356] "Generic (PLEG): container finished" podID="d93ce281-4e29-4317-a841-8f530d9dc34d" containerID="31733c218f04bb1ec6eb3b7afb82bf47bb11b79e6f751dec517926af8a9a3d89" exitCode=0 Jul 14 21:43:59.953813 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:43:59.953768 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" event={"ID":"d93ce281-4e29-4317-a841-8f530d9dc34d","Type":"ContainerDied","Data":"31733c218f04bb1ec6eb3b7afb82bf47bb11b79e6f751dec517926af8a9a3d89"} Jul 14 21:44:00.086513 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:00.086487 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" Jul 14 21:44:00.125534 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:00.125503 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7z8vw\" (UniqueName: \"kubernetes.io/projected/3446180e-7b30-45e7-a5e9-118a1384d1e7-kube-api-access-7z8vw\") pod \"3446180e-7b30-45e7-a5e9-118a1384d1e7\" (UID: \"3446180e-7b30-45e7-a5e9-118a1384d1e7\") " Jul 14 21:44:00.125716 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:00.125591 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3446180e-7b30-45e7-a5e9-118a1384d1e7-util\") pod \"3446180e-7b30-45e7-a5e9-118a1384d1e7\" (UID: \"3446180e-7b30-45e7-a5e9-118a1384d1e7\") " Jul 14 21:44:00.125716 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:00.125632 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3446180e-7b30-45e7-a5e9-118a1384d1e7-bundle\") pod \"3446180e-7b30-45e7-a5e9-118a1384d1e7\" (UID: \"3446180e-7b30-45e7-a5e9-118a1384d1e7\") " Jul 14 21:44:00.126305 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:00.126254 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3446180e-7b30-45e7-a5e9-118a1384d1e7-bundle" (OuterVolumeSpecName: "bundle") pod "3446180e-7b30-45e7-a5e9-118a1384d1e7" (UID: "3446180e-7b30-45e7-a5e9-118a1384d1e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:44:00.127881 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:00.127853 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3446180e-7b30-45e7-a5e9-118a1384d1e7-kube-api-access-7z8vw" (OuterVolumeSpecName: "kube-api-access-7z8vw") pod "3446180e-7b30-45e7-a5e9-118a1384d1e7" (UID: "3446180e-7b30-45e7-a5e9-118a1384d1e7"). InnerVolumeSpecName "kube-api-access-7z8vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:44:00.131272 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:00.131243 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3446180e-7b30-45e7-a5e9-118a1384d1e7-util" (OuterVolumeSpecName: "util") pod "3446180e-7b30-45e7-a5e9-118a1384d1e7" (UID: "3446180e-7b30-45e7-a5e9-118a1384d1e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:44:00.226836 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:00.226738 2552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3446180e-7b30-45e7-a5e9-118a1384d1e7-bundle\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:44:00.226836 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:00.226770 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z8vw\" (UniqueName: \"kubernetes.io/projected/3446180e-7b30-45e7-a5e9-118a1384d1e7-kube-api-access-7z8vw\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:44:00.226836 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:00.226783 2552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3446180e-7b30-45e7-a5e9-118a1384d1e7-util\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:44:00.962004 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:00.961966 2552 generic.go:356] "Generic (PLEG): container finished" podID="c63ce66e-682c-4e21-ac5b-b7d748d66190" containerID="9c8b27d7d6f0308bb6a15c5df8554a920cb894e909098d12ddcbb587d152f0aa" exitCode=0 Jul 14 21:44:00.962425 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:00.962048 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" event={"ID":"c63ce66e-682c-4e21-ac5b-b7d748d66190","Type":"ContainerDied","Data":"9c8b27d7d6f0308bb6a15c5df8554a920cb894e909098d12ddcbb587d152f0aa"} Jul 14 21:44:00.963801 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:00.963782 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" Jul 14 21:44:00.963801 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:00.963788 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9" event={"ID":"3446180e-7b30-45e7-a5e9-118a1384d1e7","Type":"ContainerDied","Data":"0f57c453c8e7ebbfaa12124f6af41eba0953601e2fb296532eecca8584fc80aa"} Jul 14 21:44:00.963961 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:00.963819 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f57c453c8e7ebbfaa12124f6af41eba0953601e2fb296532eecca8584fc80aa" Jul 14 21:44:00.965703 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:00.965677 2552 generic.go:356] "Generic (PLEG): container finished" podID="00780a1c-747c-41fa-a237-16fd726b14d3" containerID="0c30877ebf3e8b8d0d88b1fde4a0dd4f017cfa851bacba8e72c1052fe2c6196e" exitCode=0 Jul 14 21:44:00.965793 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:00.965757 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" event={"ID":"00780a1c-747c-41fa-a237-16fd726b14d3","Type":"ContainerDied","Data":"0c30877ebf3e8b8d0d88b1fde4a0dd4f017cfa851bacba8e72c1052fe2c6196e"} Jul 14 21:44:01.108006 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:01.107979 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" Jul 14 21:44:01.236023 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:01.235919 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d93ce281-4e29-4317-a841-8f530d9dc34d-bundle\") pod \"d93ce281-4e29-4317-a841-8f530d9dc34d\" (UID: \"d93ce281-4e29-4317-a841-8f530d9dc34d\") " Jul 14 21:44:01.236023 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:01.236001 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhkvr\" (UniqueName: \"kubernetes.io/projected/d93ce281-4e29-4317-a841-8f530d9dc34d-kube-api-access-rhkvr\") pod \"d93ce281-4e29-4317-a841-8f530d9dc34d\" (UID: \"d93ce281-4e29-4317-a841-8f530d9dc34d\") " Jul 14 21:44:01.236307 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:01.236057 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d93ce281-4e29-4317-a841-8f530d9dc34d-util\") pod \"d93ce281-4e29-4317-a841-8f530d9dc34d\" (UID: \"d93ce281-4e29-4317-a841-8f530d9dc34d\") " Jul 14 21:44:01.236528 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:01.236496 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93ce281-4e29-4317-a841-8f530d9dc34d-bundle" (OuterVolumeSpecName: "bundle") pod "d93ce281-4e29-4317-a841-8f530d9dc34d" (UID: "d93ce281-4e29-4317-a841-8f530d9dc34d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:44:01.238205 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:01.238169 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93ce281-4e29-4317-a841-8f530d9dc34d-kube-api-access-rhkvr" (OuterVolumeSpecName: "kube-api-access-rhkvr") pod "d93ce281-4e29-4317-a841-8f530d9dc34d" (UID: "d93ce281-4e29-4317-a841-8f530d9dc34d"). InnerVolumeSpecName "kube-api-access-rhkvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:44:01.241566 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:01.241543 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93ce281-4e29-4317-a841-8f530d9dc34d-util" (OuterVolumeSpecName: "util") pod "d93ce281-4e29-4317-a841-8f530d9dc34d" (UID: "d93ce281-4e29-4317-a841-8f530d9dc34d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:44:01.337548 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:01.337506 2552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d93ce281-4e29-4317-a841-8f530d9dc34d-bundle\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:44:01.337548 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:01.337539 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhkvr\" (UniqueName: \"kubernetes.io/projected/d93ce281-4e29-4317-a841-8f530d9dc34d-kube-api-access-rhkvr\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:44:01.337548 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:01.337552 2552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d93ce281-4e29-4317-a841-8f530d9dc34d-util\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:44:01.972998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:01.972971 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" Jul 14 21:44:01.973482 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:01.972996 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v" event={"ID":"d93ce281-4e29-4317-a841-8f530d9dc34d","Type":"ContainerDied","Data":"b397e43b95dd9ac953ee004674ec2b061d9dc6916db2555705593e5c26fe9ca5"} Jul 14 21:44:01.973482 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:01.973042 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b397e43b95dd9ac953ee004674ec2b061d9dc6916db2555705593e5c26fe9ca5" Jul 14 21:44:02.112779 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.112750 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" Jul 14 21:44:02.148514 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.148491 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" Jul 14 21:44:02.245137 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.245043 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00780a1c-747c-41fa-a237-16fd726b14d3-util\") pod \"00780a1c-747c-41fa-a237-16fd726b14d3\" (UID: \"00780a1c-747c-41fa-a237-16fd726b14d3\") " Jul 14 21:44:02.245137 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.245090 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4msh2\" (UniqueName: \"kubernetes.io/projected/c63ce66e-682c-4e21-ac5b-b7d748d66190-kube-api-access-4msh2\") pod \"c63ce66e-682c-4e21-ac5b-b7d748d66190\" (UID: \"c63ce66e-682c-4e21-ac5b-b7d748d66190\") " Jul 14 21:44:02.245137 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.245109 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c63ce66e-682c-4e21-ac5b-b7d748d66190-bundle\") pod \"c63ce66e-682c-4e21-ac5b-b7d748d66190\" (UID: \"c63ce66e-682c-4e21-ac5b-b7d748d66190\") " Jul 14 21:44:02.245137 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.245126 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c63ce66e-682c-4e21-ac5b-b7d748d66190-util\") pod \"c63ce66e-682c-4e21-ac5b-b7d748d66190\" (UID: \"c63ce66e-682c-4e21-ac5b-b7d748d66190\") " Jul 14 21:44:02.245529 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.245176 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00780a1c-747c-41fa-a237-16fd726b14d3-bundle\") pod \"00780a1c-747c-41fa-a237-16fd726b14d3\" (UID: \"00780a1c-747c-41fa-a237-16fd726b14d3\") " Jul 14 21:44:02.245529 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.245261 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbrp7\" (UniqueName: \"kubernetes.io/projected/00780a1c-747c-41fa-a237-16fd726b14d3-kube-api-access-qbrp7\") pod \"00780a1c-747c-41fa-a237-16fd726b14d3\" (UID: \"00780a1c-747c-41fa-a237-16fd726b14d3\") " Jul 14 21:44:02.245757 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.245735 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63ce66e-682c-4e21-ac5b-b7d748d66190-bundle" (OuterVolumeSpecName: "bundle") pod "c63ce66e-682c-4e21-ac5b-b7d748d66190" (UID: "c63ce66e-682c-4e21-ac5b-b7d748d66190"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:44:02.245994 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.245964 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00780a1c-747c-41fa-a237-16fd726b14d3-bundle" (OuterVolumeSpecName: "bundle") pod "00780a1c-747c-41fa-a237-16fd726b14d3" (UID: "00780a1c-747c-41fa-a237-16fd726b14d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:44:02.247502 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.247466 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63ce66e-682c-4e21-ac5b-b7d748d66190-kube-api-access-4msh2" (OuterVolumeSpecName: "kube-api-access-4msh2") pod "c63ce66e-682c-4e21-ac5b-b7d748d66190" (UID: "c63ce66e-682c-4e21-ac5b-b7d748d66190"). InnerVolumeSpecName "kube-api-access-4msh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:44:02.247841 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.247814 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00780a1c-747c-41fa-a237-16fd726b14d3-kube-api-access-qbrp7" (OuterVolumeSpecName: "kube-api-access-qbrp7") pod "00780a1c-747c-41fa-a237-16fd726b14d3" (UID: "00780a1c-747c-41fa-a237-16fd726b14d3"). InnerVolumeSpecName "kube-api-access-qbrp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:44:02.251346 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.251319 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63ce66e-682c-4e21-ac5b-b7d748d66190-util" (OuterVolumeSpecName: "util") pod "c63ce66e-682c-4e21-ac5b-b7d748d66190" (UID: "c63ce66e-682c-4e21-ac5b-b7d748d66190"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:44:02.251869 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.251847 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00780a1c-747c-41fa-a237-16fd726b14d3-util" (OuterVolumeSpecName: "util") pod "00780a1c-747c-41fa-a237-16fd726b14d3" (UID: "00780a1c-747c-41fa-a237-16fd726b14d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 21:44:02.346610 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.346554 2552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00780a1c-747c-41fa-a237-16fd726b14d3-util\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:44:02.346610 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.346605 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4msh2\" (UniqueName: \"kubernetes.io/projected/c63ce66e-682c-4e21-ac5b-b7d748d66190-kube-api-access-4msh2\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:44:02.346610 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.346618 2552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c63ce66e-682c-4e21-ac5b-b7d748d66190-bundle\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:44:02.346610 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.346626 2552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c63ce66e-682c-4e21-ac5b-b7d748d66190-util\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:44:02.346891 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.346634 2552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00780a1c-747c-41fa-a237-16fd726b14d3-bundle\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:44:02.346891 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.346641 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbrp7\" (UniqueName: \"kubernetes.io/projected/00780a1c-747c-41fa-a237-16fd726b14d3-kube-api-access-qbrp7\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:44:02.979681 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.979647 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" Jul 14 21:44:02.979681 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.979671 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g" event={"ID":"c63ce66e-682c-4e21-ac5b-b7d748d66190","Type":"ContainerDied","Data":"924a1f33ffaa3f4f3dafcb571b07848a9ae90e5e9df8aded39cc86f3d3e9a4fa"} Jul 14 21:44:02.980093 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.979721 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="924a1f33ffaa3f4f3dafcb571b07848a9ae90e5e9df8aded39cc86f3d3e9a4fa" Jul 14 21:44:02.981665 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.981641 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" event={"ID":"00780a1c-747c-41fa-a237-16fd726b14d3","Type":"ContainerDied","Data":"a87a473a7f2474ceecdde3e0ee7d902ea1b743d4fa6b73a84ccce30d175afa80"} Jul 14 21:44:02.981795 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.981673 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx" Jul 14 21:44:02.981795 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:02.981681 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a87a473a7f2474ceecdde3e0ee7d902ea1b743d4fa6b73a84ccce30d175afa80" Jul 14 21:44:09.431891 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:09.431849 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:13.015866 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:13.015833 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:14.388140 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:14.388100 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:14.785565 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:14.785474 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:15.816464 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:15.816427 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:15.911419 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:15.911386 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:17.385661 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:17.385619 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:17.480941 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:17.480904 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:19.587820 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:19.587784 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:21.413973 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:21.413931 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:23.036826 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.036790 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zfz7t"] Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037105 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3446180e-7b30-45e7-a5e9-118a1384d1e7" containerName="extract" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037116 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3446180e-7b30-45e7-a5e9-118a1384d1e7" containerName="extract" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037123 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c63ce66e-682c-4e21-ac5b-b7d748d66190" containerName="extract" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037130 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c63ce66e-682c-4e21-ac5b-b7d748d66190" containerName="extract" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037140 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00780a1c-747c-41fa-a237-16fd726b14d3" containerName="extract" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037146 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="00780a1c-747c-41fa-a237-16fd726b14d3" containerName="extract" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037153 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c63ce66e-682c-4e21-ac5b-b7d748d66190" containerName="pull" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037158 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c63ce66e-682c-4e21-ac5b-b7d748d66190" containerName="pull" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037169 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d93ce281-4e29-4317-a841-8f530d9dc34d" containerName="extract" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037174 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d93ce281-4e29-4317-a841-8f530d9dc34d" containerName="extract" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037183 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3446180e-7b30-45e7-a5e9-118a1384d1e7" containerName="util" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037202 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3446180e-7b30-45e7-a5e9-118a1384d1e7" containerName="util" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037208 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d93ce281-4e29-4317-a841-8f530d9dc34d" containerName="util" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037213 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d93ce281-4e29-4317-a841-8f530d9dc34d" containerName="util" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037219 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00780a1c-747c-41fa-a237-16fd726b14d3" containerName="pull" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037224 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="00780a1c-747c-41fa-a237-16fd726b14d3" containerName="pull" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037231 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00780a1c-747c-41fa-a237-16fd726b14d3" containerName="util" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037236 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="00780a1c-747c-41fa-a237-16fd726b14d3" containerName="util" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037242 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d93ce281-4e29-4317-a841-8f530d9dc34d" containerName="pull" Jul 14 21:44:23.037237 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037247 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d93ce281-4e29-4317-a841-8f530d9dc34d" containerName="pull" Jul 14 21:44:23.037846 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037257 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c63ce66e-682c-4e21-ac5b-b7d748d66190" containerName="util" Jul 14 21:44:23.037846 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037262 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c63ce66e-682c-4e21-ac5b-b7d748d66190" containerName="util" Jul 14 21:44:23.037846 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037268 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3446180e-7b30-45e7-a5e9-118a1384d1e7" containerName="pull" Jul 14 21:44:23.037846 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037273 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3446180e-7b30-45e7-a5e9-118a1384d1e7" containerName="pull" Jul 14 21:44:23.037846 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037324 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3446180e-7b30-45e7-a5e9-118a1384d1e7" containerName="extract" Jul 14 21:44:23.037846 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037333 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c63ce66e-682c-4e21-ac5b-b7d748d66190" containerName="extract" Jul 14 21:44:23.037846 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037338 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d93ce281-4e29-4317-a841-8f530d9dc34d" containerName="extract" Jul 14 21:44:23.037846 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.037344 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="00780a1c-747c-41fa-a237-16fd726b14d3" containerName="extract" Jul 14 21:44:23.039893 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.039859 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:23.040315 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.040298 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfz7t" Jul 14 21:44:23.043102 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.043076 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 14 21:44:23.043102 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.043101 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 14 21:44:23.043296 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.043128 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 21:44:23.043296 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.043184 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 21:44:23.044142 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.044127 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mgtcm\"" Jul 14 21:44:23.051841 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.051815 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zfz7t"] Jul 14 21:44:23.198213 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.198160 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:23.229270 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.229230 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqnm9\" (UniqueName: \"kubernetes.io/projected/e30d8d46-d5e4-4a96-b261-8a2130c887ac-kube-api-access-zqnm9\") pod \"kuadrant-console-plugin-66b5774985-zfz7t\" (UID: \"e30d8d46-d5e4-4a96-b261-8a2130c887ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfz7t" Jul 14 21:44:23.229449 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.229291 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e30d8d46-d5e4-4a96-b261-8a2130c887ac-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zfz7t\" (UID: \"e30d8d46-d5e4-4a96-b261-8a2130c887ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfz7t" Jul 14 21:44:23.229449 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.229346 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e30d8d46-d5e4-4a96-b261-8a2130c887ac-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zfz7t\" (UID: \"e30d8d46-d5e4-4a96-b261-8a2130c887ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfz7t" Jul 14 21:44:23.297536 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.297447 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:23.330825 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.330786 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e30d8d46-d5e4-4a96-b261-8a2130c887ac-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zfz7t\" (UID: \"e30d8d46-d5e4-4a96-b261-8a2130c887ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfz7t" Jul 14 21:44:23.331010 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.330858 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e30d8d46-d5e4-4a96-b261-8a2130c887ac-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zfz7t\" (UID: \"e30d8d46-d5e4-4a96-b261-8a2130c887ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfz7t" Jul 14 21:44:23.331010 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.330920 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqnm9\" (UniqueName: \"kubernetes.io/projected/e30d8d46-d5e4-4a96-b261-8a2130c887ac-kube-api-access-zqnm9\") pod \"kuadrant-console-plugin-66b5774985-zfz7t\" (UID: \"e30d8d46-d5e4-4a96-b261-8a2130c887ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfz7t" Jul 14 21:44:23.331639 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.331616 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e30d8d46-d5e4-4a96-b261-8a2130c887ac-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zfz7t\" (UID: \"e30d8d46-d5e4-4a96-b261-8a2130c887ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfz7t" Jul 14 21:44:23.333325 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.333307 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e30d8d46-d5e4-4a96-b261-8a2130c887ac-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zfz7t\" (UID: \"e30d8d46-d5e4-4a96-b261-8a2130c887ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfz7t" Jul 14 21:44:23.340907 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.340886 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqnm9\" (UniqueName: \"kubernetes.io/projected/e30d8d46-d5e4-4a96-b261-8a2130c887ac-kube-api-access-zqnm9\") pod \"kuadrant-console-plugin-66b5774985-zfz7t\" (UID: \"e30d8d46-d5e4-4a96-b261-8a2130c887ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfz7t" Jul 14 21:44:23.349297 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.349277 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfz7t" Jul 14 21:44:23.490185 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:23.490147 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zfz7t"] Jul 14 21:44:23.494796 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:44:23.494749 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode30d8d46_d5e4_4a96_b261_8a2130c887ac.slice/crio-9cd0ead6380a22a05d3b5f932bfd60f9db4c89316e7f6205fd38fc75ecd77817 WatchSource:0}: Error finding container 9cd0ead6380a22a05d3b5f932bfd60f9db4c89316e7f6205fd38fc75ecd77817: Status 404 returned error can't find the container with id 9cd0ead6380a22a05d3b5f932bfd60f9db4c89316e7f6205fd38fc75ecd77817 Jul 14 21:44:24.087236 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:24.087182 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfz7t" event={"ID":"e30d8d46-d5e4-4a96-b261-8a2130c887ac","Type":"ContainerStarted","Data":"9cd0ead6380a22a05d3b5f932bfd60f9db4c89316e7f6205fd38fc75ecd77817"} Jul 14 21:44:25.875290 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:25.875224 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:32.450856 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:32.450816 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:32.550748 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:32.550710 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:32.653665 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:32.653624 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:34.887184 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:34.887142 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:34.994282 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:34.994242 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:40.762284 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:40.762160 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:40.866372 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:40.866323 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:45.976305 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:45.976235 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:46.080604 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:46.080570 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:48.233013 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:48.232974 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfz7t" event={"ID":"e30d8d46-d5e4-4a96-b261-8a2130c887ac","Type":"ContainerStarted","Data":"de7c9b3bbfb66432e1f2b2918c7520f6f0bded8a8cac6e3f828e5dc6f34fc2eb"} Jul 14 21:44:48.254020 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:48.253945 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfz7t" podStartSLOduration=1.575491967 podStartE2EDuration="25.253928668s" totalImagesPullingTime="23.678436701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:44:48.252056325 +0000 UTC m=+454.790762011" watchObservedRunningTime="2026-07-14 21:44:48.253928668 +0000 UTC m=+454.792634340" Jul 14 21:44:55.972160 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:55.972121 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:56.068186 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:56.068138 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:56.533030 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:56.532993 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:56.634507 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:56.634473 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:58.412938 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:58.412898 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:59.727794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:59.727756 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:44:59.792864 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:44:59.792812 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:03.009660 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:03.009621 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:03.396985 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:03.396950 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:03.823951 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:03.823856 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:03.925733 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:03.925694 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:05.266862 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:05.266811 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:05.428586 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:05.428541 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:05.522115 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:05.522012 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:06.248647 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:06.248610 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-l5xzp"] Jul 14 21:45:06.332619 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:06.332580 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:06.357609 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:06.357570 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-l5xzp"] Jul 14 21:45:06.357809 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:06.357714 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l5xzp" Jul 14 21:45:06.362225 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:06.362200 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7gdxl\"" Jul 14 21:45:06.427154 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:06.427118 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:06.507947 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:06.507849 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwf7c\" (UniqueName: \"kubernetes.io/projected/c92d3e1a-04db-477c-ae49-f48c4dc7f33e-kube-api-access-rwf7c\") pod \"authorino-58cd955675-l5xzp\" (UID: \"c92d3e1a-04db-477c-ae49-f48c4dc7f33e\") " pod="kuadrant-system/authorino-58cd955675-l5xzp" Jul 14 21:45:06.609164 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:06.609124 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwf7c\" (UniqueName: \"kubernetes.io/projected/c92d3e1a-04db-477c-ae49-f48c4dc7f33e-kube-api-access-rwf7c\") pod \"authorino-58cd955675-l5xzp\" (UID: \"c92d3e1a-04db-477c-ae49-f48c4dc7f33e\") " pod="kuadrant-system/authorino-58cd955675-l5xzp" Jul 14 21:45:06.620297 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:06.620269 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwf7c\" (UniqueName: \"kubernetes.io/projected/c92d3e1a-04db-477c-ae49-f48c4dc7f33e-kube-api-access-rwf7c\") pod \"authorino-58cd955675-l5xzp\" (UID: \"c92d3e1a-04db-477c-ae49-f48c4dc7f33e\") " pod="kuadrant-system/authorino-58cd955675-l5xzp" Jul 14 21:45:06.667308 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:06.667264 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l5xzp" Jul 14 21:45:06.903315 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:45:06.903274 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc92d3e1a_04db_477c_ae49_f48c4dc7f33e.slice/crio-b5c1be9964200a3638cec31e0395e8a886b3d515536b7872a6f901d71eed49da WatchSource:0}: Error finding container b5c1be9964200a3638cec31e0395e8a886b3d515536b7872a6f901d71eed49da: Status 404 returned error can't find the container with id b5c1be9964200a3638cec31e0395e8a886b3d515536b7872a6f901d71eed49da Jul 14 21:45:06.904167 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:06.904141 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-l5xzp"] Jul 14 21:45:07.002660 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:07.002626 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-2mthm"] Jul 14 21:45:07.012980 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:07.012954 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-2mthm" Jul 14 21:45:07.026996 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:07.026966 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2mthm"] Jul 14 21:45:07.113885 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:07.113840 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8vgh\" (UniqueName: \"kubernetes.io/projected/c2fec568-4024-4ae1-b11d-0c325f291066-kube-api-access-c8vgh\") pod \"authorino-75d556bd77-2mthm\" (UID: \"c2fec568-4024-4ae1-b11d-0c325f291066\") " pod="kuadrant-system/authorino-75d556bd77-2mthm" Jul 14 21:45:07.215466 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:07.215429 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8vgh\" (UniqueName: \"kubernetes.io/projected/c2fec568-4024-4ae1-b11d-0c325f291066-kube-api-access-c8vgh\") pod \"authorino-75d556bd77-2mthm\" (UID: \"c2fec568-4024-4ae1-b11d-0c325f291066\") " pod="kuadrant-system/authorino-75d556bd77-2mthm" Jul 14 21:45:07.251795 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:07.251766 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8vgh\" (UniqueName: \"kubernetes.io/projected/c2fec568-4024-4ae1-b11d-0c325f291066-kube-api-access-c8vgh\") pod \"authorino-75d556bd77-2mthm\" (UID: \"c2fec568-4024-4ae1-b11d-0c325f291066\") " pod="kuadrant-system/authorino-75d556bd77-2mthm" Jul 14 21:45:07.322353 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:07.322325 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-2mthm" Jul 14 21:45:07.326001 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:07.325971 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l5xzp" event={"ID":"c92d3e1a-04db-477c-ae49-f48c4dc7f33e","Type":"ContainerStarted","Data":"b5c1be9964200a3638cec31e0395e8a886b3d515536b7872a6f901d71eed49da"} Jul 14 21:45:07.489240 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:07.489210 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2mthm"] Jul 14 21:45:07.490633 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:45:07.490602 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2fec568_4024_4ae1_b11d_0c325f291066.slice/crio-3995d9f26623f562c99e581ffd253678fbf9374a415df00fdbe6dfc5cf609d73 WatchSource:0}: Error finding container 3995d9f26623f562c99e581ffd253678fbf9374a415df00fdbe6dfc5cf609d73: Status 404 returned error can't find the container with id 3995d9f26623f562c99e581ffd253678fbf9374a415df00fdbe6dfc5cf609d73 Jul 14 21:45:07.815528 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:07.815435 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:07.941833 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:07.941801 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:08.334578 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:08.334537 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-2mthm" event={"ID":"c2fec568-4024-4ae1-b11d-0c325f291066","Type":"ContainerStarted","Data":"3995d9f26623f562c99e581ffd253678fbf9374a415df00fdbe6dfc5cf609d73"} Jul 14 21:45:12.363829 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:12.363780 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l5xzp" event={"ID":"c92d3e1a-04db-477c-ae49-f48c4dc7f33e","Type":"ContainerStarted","Data":"a2d090bb02accc899e6ca3157565c1c94cd1211a745d3c66a6351f0f0045f777"} Jul 14 21:45:12.365551 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:12.365525 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-2mthm" event={"ID":"c2fec568-4024-4ae1-b11d-0c325f291066","Type":"ContainerStarted","Data":"1dbee9441a03f5ca055729b63ebcdc8cea64f904cc31a16491a36d8800f33634"} Jul 14 21:45:12.382434 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:12.382337 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-l5xzp" podStartSLOduration=1.15780703 podStartE2EDuration="6.382320847s" totalImagesPullingTime="5.224513817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:45:12.380845455 +0000 UTC m=+478.919551128" watchObservedRunningTime="2026-07-14 21:45:12.382320847 +0000 UTC m=+478.921026571" Jul 14 21:45:12.398587 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:12.398530 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-2mthm" podStartSLOduration=1.748867578 podStartE2EDuration="6.398515463s" totalImagesPullingTime="4.649647885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:45:12.396827326 +0000 UTC m=+478.935533008" watchObservedRunningTime="2026-07-14 21:45:12.398515463 +0000 UTC m=+478.937221135" Jul 14 21:45:12.425006 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:12.424970 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-l5xzp"] Jul 14 21:45:14.375685 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:14.375623 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-l5xzp" podUID="c92d3e1a-04db-477c-ae49-f48c4dc7f33e" containerName="authorino" containerID="cri-o://a2d090bb02accc899e6ca3157565c1c94cd1211a745d3c66a6351f0f0045f777" gracePeriod=30 Jul 14 21:45:14.631692 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:14.631631 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l5xzp" Jul 14 21:45:14.788435 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:14.788402 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwf7c\" (UniqueName: \"kubernetes.io/projected/c92d3e1a-04db-477c-ae49-f48c4dc7f33e-kube-api-access-rwf7c\") pod \"c92d3e1a-04db-477c-ae49-f48c4dc7f33e\" (UID: \"c92d3e1a-04db-477c-ae49-f48c4dc7f33e\") " Jul 14 21:45:14.796034 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:14.795778 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92d3e1a-04db-477c-ae49-f48c4dc7f33e-kube-api-access-rwf7c" (OuterVolumeSpecName: "kube-api-access-rwf7c") pod "c92d3e1a-04db-477c-ae49-f48c4dc7f33e" (UID: "c92d3e1a-04db-477c-ae49-f48c4dc7f33e"). InnerVolumeSpecName "kube-api-access-rwf7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:45:14.889956 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:14.889863 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwf7c\" (UniqueName: \"kubernetes.io/projected/c92d3e1a-04db-477c-ae49-f48c4dc7f33e-kube-api-access-rwf7c\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:45:15.381271 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:15.381232 2552 generic.go:356] "Generic (PLEG): container finished" podID="c92d3e1a-04db-477c-ae49-f48c4dc7f33e" containerID="a2d090bb02accc899e6ca3157565c1c94cd1211a745d3c66a6351f0f0045f777" exitCode=0 Jul 14 21:45:15.381710 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:15.381288 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l5xzp" Jul 14 21:45:15.381710 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:15.381320 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l5xzp" event={"ID":"c92d3e1a-04db-477c-ae49-f48c4dc7f33e","Type":"ContainerDied","Data":"a2d090bb02accc899e6ca3157565c1c94cd1211a745d3c66a6351f0f0045f777"} Jul 14 21:45:15.381710 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:15.381365 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l5xzp" event={"ID":"c92d3e1a-04db-477c-ae49-f48c4dc7f33e","Type":"ContainerDied","Data":"b5c1be9964200a3638cec31e0395e8a886b3d515536b7872a6f901d71eed49da"} Jul 14 21:45:15.381710 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:15.381383 2552 scope.go:117] "RemoveContainer" containerID="a2d090bb02accc899e6ca3157565c1c94cd1211a745d3c66a6351f0f0045f777" Jul 14 21:45:15.390701 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:15.390683 2552 scope.go:117] "RemoveContainer" containerID="a2d090bb02accc899e6ca3157565c1c94cd1211a745d3c66a6351f0f0045f777" Jul 14 21:45:15.390977 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:45:15.390956 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d090bb02accc899e6ca3157565c1c94cd1211a745d3c66a6351f0f0045f777\": container with ID starting with a2d090bb02accc899e6ca3157565c1c94cd1211a745d3c66a6351f0f0045f777 not found: ID does not exist" containerID="a2d090bb02accc899e6ca3157565c1c94cd1211a745d3c66a6351f0f0045f777" Jul 14 21:45:15.391041 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:15.390986 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d090bb02accc899e6ca3157565c1c94cd1211a745d3c66a6351f0f0045f777"} err="failed to get container status \"a2d090bb02accc899e6ca3157565c1c94cd1211a745d3c66a6351f0f0045f777\": rpc error: code = NotFound desc = could not find container \"a2d090bb02accc899e6ca3157565c1c94cd1211a745d3c66a6351f0f0045f777\": container with ID starting with a2d090bb02accc899e6ca3157565c1c94cd1211a745d3c66a6351f0f0045f777 not found: ID does not exist" Jul 14 21:45:15.403798 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:15.403774 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-l5xzp"] Jul 14 21:45:15.410647 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:15.410623 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-l5xzp"] Jul 14 21:45:15.991632 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:15.991601 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c92d3e1a-04db-477c-ae49-f48c4dc7f33e" path="/var/lib/kubelet/pods/c92d3e1a-04db-477c-ae49-f48c4dc7f33e/volumes" Jul 14 21:45:17.264015 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:17.263979 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:17.369773 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:17.369721 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:18.232667 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:18.232616 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:19.190714 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:19.190673 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:19.536986 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:19.536889 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:19.636792 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:19.636756 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:24.157048 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:24.157010 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:24.259469 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:24.259431 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:25.186291 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:25.186248 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:26.983538 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:26.983505 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:36.293781 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:36.293737 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-664d86cc8-wmwlp"] Jul 14 21:45:36.454929 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:36.454888 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:36.551268 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:36.551143 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:36.622042 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:36.622005 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:38.630890 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:38.630832 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:38.834843 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:38.834793 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:38.883915 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:38.883826 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjpj6"] Jul 14 21:45:38.884179 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:38.884166 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c92d3e1a-04db-477c-ae49-f48c4dc7f33e" containerName="authorino" Jul 14 21:45:38.884264 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:38.884181 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c92d3e1a-04db-477c-ae49-f48c4dc7f33e" containerName="authorino" Jul 14 21:45:38.884264 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:38.884251 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c92d3e1a-04db-477c-ae49-f48c4dc7f33e" containerName="authorino" Jul 14 21:45:38.886093 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:38.886075 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sjpj6" Jul 14 21:45:38.908936 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:38.908900 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjpj6"] Jul 14 21:45:38.973288 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:38.973249 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmcjq\" (UniqueName: \"kubernetes.io/projected/fd9fc061-0dd3-403d-825b-2ecf6f829512-kube-api-access-gmcjq\") pod \"authorino-5d4b67c446-sjpj6\" (UID: \"fd9fc061-0dd3-403d-825b-2ecf6f829512\") " pod="kuadrant-system/authorino-5d4b67c446-sjpj6" Jul 14 21:45:39.074235 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.074165 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmcjq\" (UniqueName: \"kubernetes.io/projected/fd9fc061-0dd3-403d-825b-2ecf6f829512-kube-api-access-gmcjq\") pod \"authorino-5d4b67c446-sjpj6\" (UID: \"fd9fc061-0dd3-403d-825b-2ecf6f829512\") " pod="kuadrant-system/authorino-5d4b67c446-sjpj6" Jul 14 21:45:39.086828 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.086794 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmcjq\" (UniqueName: \"kubernetes.io/projected/fd9fc061-0dd3-403d-825b-2ecf6f829512-kube-api-access-gmcjq\") pod \"authorino-5d4b67c446-sjpj6\" (UID: \"fd9fc061-0dd3-403d-825b-2ecf6f829512\") " pod="kuadrant-system/authorino-5d4b67c446-sjpj6" Jul 14 21:45:39.164311 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.164213 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjpj6"] Jul 14 21:45:39.164527 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.164508 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sjpj6" Jul 14 21:45:39.195264 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.195222 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-865b878667-gxdjs"] Jul 14 21:45:39.200926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.200890 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-865b878667-gxdjs" Jul 14 21:45:39.208933 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.208901 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-865b878667-gxdjs"] Jul 14 21:45:39.275977 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.275934 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpgd9\" (UniqueName: \"kubernetes.io/projected/312aa157-e1b4-42fe-b152-401e9a0424cd-kube-api-access-dpgd9\") pod \"authorino-865b878667-gxdjs\" (UID: \"312aa157-e1b4-42fe-b152-401e9a0424cd\") " pod="kuadrant-system/authorino-865b878667-gxdjs" Jul 14 21:45:39.306840 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.306809 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjpj6"] Jul 14 21:45:39.308693 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:45:39.308659 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd9fc061_0dd3_403d_825b_2ecf6f829512.slice/crio-43256cb10f5378850048c62c839e100427950186501b9c65773752631a12419d WatchSource:0}: Error finding container 43256cb10f5378850048c62c839e100427950186501b9c65773752631a12419d: Status 404 returned error can't find the container with id 43256cb10f5378850048c62c839e100427950186501b9c65773752631a12419d Jul 14 21:45:39.376611 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.376570 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpgd9\" (UniqueName: \"kubernetes.io/projected/312aa157-e1b4-42fe-b152-401e9a0424cd-kube-api-access-dpgd9\") pod \"authorino-865b878667-gxdjs\" (UID: \"312aa157-e1b4-42fe-b152-401e9a0424cd\") " pod="kuadrant-system/authorino-865b878667-gxdjs" Jul 14 21:45:39.387321 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.387293 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpgd9\" (UniqueName: \"kubernetes.io/projected/312aa157-e1b4-42fe-b152-401e9a0424cd-kube-api-access-dpgd9\") pod \"authorino-865b878667-gxdjs\" (UID: \"312aa157-e1b4-42fe-b152-401e9a0424cd\") " pod="kuadrant-system/authorino-865b878667-gxdjs" Jul 14 21:45:39.461526 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.461439 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-865b878667-gxdjs"] Jul 14 21:45:39.461718 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.461705 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-865b878667-gxdjs" Jul 14 21:45:39.506835 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.506791 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sjpj6" event={"ID":"fd9fc061-0dd3-403d-825b-2ecf6f829512","Type":"ContainerStarted","Data":"43256cb10f5378850048c62c839e100427950186501b9c65773752631a12419d"} Jul 14 21:45:39.534224 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.533904 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c7d9f96f9-kts62"] Jul 14 21:45:39.537280 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.537251 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c7d9f96f9-kts62" Jul 14 21:45:39.540704 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.540499 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 14 21:45:39.547050 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.546997 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c7d9f96f9-kts62"] Jul 14 21:45:39.578373 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.578339 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d29997d2-a75c-4d10-a812-57d36aa57232-tls-cert\") pod \"authorino-c7d9f96f9-kts62\" (UID: \"d29997d2-a75c-4d10-a812-57d36aa57232\") " pod="kuadrant-system/authorino-c7d9f96f9-kts62" Jul 14 21:45:39.578543 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.578421 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7mmw\" (UniqueName: \"kubernetes.io/projected/d29997d2-a75c-4d10-a812-57d36aa57232-kube-api-access-r7mmw\") pod \"authorino-c7d9f96f9-kts62\" (UID: \"d29997d2-a75c-4d10-a812-57d36aa57232\") " pod="kuadrant-system/authorino-c7d9f96f9-kts62" Jul 14 21:45:39.640946 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.640919 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-865b878667-gxdjs"] Jul 14 21:45:39.643039 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:45:39.643009 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod312aa157_e1b4_42fe_b152_401e9a0424cd.slice/crio-2f1bf5d1a48e98eef8bf90e021493e77c94fb2795a872ba37639fbcaf2b1c294 WatchSource:0}: Error finding container 2f1bf5d1a48e98eef8bf90e021493e77c94fb2795a872ba37639fbcaf2b1c294: Status 404 returned error can't find the container with id 2f1bf5d1a48e98eef8bf90e021493e77c94fb2795a872ba37639fbcaf2b1c294 Jul 14 21:45:39.679241 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.679201 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d29997d2-a75c-4d10-a812-57d36aa57232-tls-cert\") pod \"authorino-c7d9f96f9-kts62\" (UID: \"d29997d2-a75c-4d10-a812-57d36aa57232\") " pod="kuadrant-system/authorino-c7d9f96f9-kts62" Jul 14 21:45:39.679241 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.679244 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7mmw\" (UniqueName: \"kubernetes.io/projected/d29997d2-a75c-4d10-a812-57d36aa57232-kube-api-access-r7mmw\") pod \"authorino-c7d9f96f9-kts62\" (UID: \"d29997d2-a75c-4d10-a812-57d36aa57232\") " pod="kuadrant-system/authorino-c7d9f96f9-kts62" Jul 14 21:45:39.681690 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.681665 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d29997d2-a75c-4d10-a812-57d36aa57232-tls-cert\") pod \"authorino-c7d9f96f9-kts62\" (UID: \"d29997d2-a75c-4d10-a812-57d36aa57232\") " pod="kuadrant-system/authorino-c7d9f96f9-kts62" Jul 14 21:45:39.689613 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.689593 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7mmw\" (UniqueName: \"kubernetes.io/projected/d29997d2-a75c-4d10-a812-57d36aa57232-kube-api-access-r7mmw\") pod \"authorino-c7d9f96f9-kts62\" (UID: \"d29997d2-a75c-4d10-a812-57d36aa57232\") " pod="kuadrant-system/authorino-c7d9f96f9-kts62" Jul 14 21:45:39.851785 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.851753 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c7d9f96f9-kts62" Jul 14 21:45:39.994422 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:39.994360 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c7d9f96f9-kts62"] Jul 14 21:45:39.996620 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:45:39.996591 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd29997d2_a75c_4d10_a812_57d36aa57232.slice/crio-b191c2ac57f548327618d2647e4c88d919cefac5850aa64d7feff95ad903b94a WatchSource:0}: Error finding container b191c2ac57f548327618d2647e4c88d919cefac5850aa64d7feff95ad903b94a: Status 404 returned error can't find the container with id b191c2ac57f548327618d2647e4c88d919cefac5850aa64d7feff95ad903b94a Jul 14 21:45:40.514770 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.514730 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c7d9f96f9-kts62" event={"ID":"d29997d2-a75c-4d10-a812-57d36aa57232","Type":"ContainerStarted","Data":"65883e917af3f478f7648498590bc127626adb7352b025917c9fb2be238f2330"} Jul 14 21:45:40.514926 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.514782 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c7d9f96f9-kts62" event={"ID":"d29997d2-a75c-4d10-a812-57d36aa57232","Type":"ContainerStarted","Data":"b191c2ac57f548327618d2647e4c88d919cefac5850aa64d7feff95ad903b94a"} Jul 14 21:45:40.516473 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.516443 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-865b878667-gxdjs" event={"ID":"312aa157-e1b4-42fe-b152-401e9a0424cd","Type":"ContainerStarted","Data":"da76609f64ca81fbccbbed36e57daee3930f246ad6506d1aea0720fceb80aa2e"} Jul 14 21:45:40.516600 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.516483 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-865b878667-gxdjs" event={"ID":"312aa157-e1b4-42fe-b152-401e9a0424cd","Type":"ContainerStarted","Data":"2f1bf5d1a48e98eef8bf90e021493e77c94fb2795a872ba37639fbcaf2b1c294"} Jul 14 21:45:40.516600 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.516498 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-865b878667-gxdjs" podUID="312aa157-e1b4-42fe-b152-401e9a0424cd" containerName="authorino" containerID="cri-o://da76609f64ca81fbccbbed36e57daee3930f246ad6506d1aea0720fceb80aa2e" gracePeriod=30 Jul 14 21:45:40.518407 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.518378 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sjpj6" event={"ID":"fd9fc061-0dd3-403d-825b-2ecf6f829512","Type":"ContainerStarted","Data":"e9d4f2afb8beed7942fd340a486277dbfb1ef3a6849c678d515c5c15aea2076b"} Jul 14 21:45:40.518516 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.518455 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-sjpj6" podUID="fd9fc061-0dd3-403d-825b-2ecf6f829512" containerName="authorino" containerID="cri-o://e9d4f2afb8beed7942fd340a486277dbfb1ef3a6849c678d515c5c15aea2076b" gracePeriod=30 Jul 14 21:45:40.545867 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.545812 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c7d9f96f9-kts62" podStartSLOduration=1.124925886 podStartE2EDuration="1.545794643s" totalImagesPullingTime="420.868757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:45:40.542572275 +0000 UTC m=+507.081277948" watchObservedRunningTime="2026-07-14 21:45:40.545794643 +0000 UTC m=+507.084500293" Jul 14 21:45:40.571814 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.571753 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-865b878667-gxdjs" podStartSLOduration=1.027738449 podStartE2EDuration="1.571737536s" totalImagesPullingTime="543.999087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:45:40.570696243 +0000 UTC m=+507.109401918" watchObservedRunningTime="2026-07-14 21:45:40.571737536 +0000 UTC m=+507.110443209" Jul 14 21:45:40.581436 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.581400 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2mthm"] Jul 14 21:45:40.581660 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.581634 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-2mthm" podUID="c2fec568-4024-4ae1-b11d-0c325f291066" containerName="authorino" containerID="cri-o://1dbee9441a03f5ca055729b63ebcdc8cea64f904cc31a16491a36d8800f33634" gracePeriod=30 Jul 14 21:45:40.595067 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.595010 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-sjpj6" podStartSLOduration=2.164726784 podStartE2EDuration="2.594992141s" totalImagesPullingTime="430.265357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:45:40.593647334 +0000 UTC m=+507.132353007" watchObservedRunningTime="2026-07-14 21:45:40.594992141 +0000 UTC m=+507.133697812" Jul 14 21:45:40.820999 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.820971 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sjpj6" Jul 14 21:45:40.863676 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.863648 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-865b878667-gxdjs" Jul 14 21:45:40.890984 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.890954 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpgd9\" (UniqueName: \"kubernetes.io/projected/312aa157-e1b4-42fe-b152-401e9a0424cd-kube-api-access-dpgd9\") pod \"312aa157-e1b4-42fe-b152-401e9a0424cd\" (UID: \"312aa157-e1b4-42fe-b152-401e9a0424cd\") " Jul 14 21:45:40.891147 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.891054 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmcjq\" (UniqueName: \"kubernetes.io/projected/fd9fc061-0dd3-403d-825b-2ecf6f829512-kube-api-access-gmcjq\") pod \"fd9fc061-0dd3-403d-825b-2ecf6f829512\" (UID: \"fd9fc061-0dd3-403d-825b-2ecf6f829512\") " Jul 14 21:45:40.893178 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.893147 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9fc061-0dd3-403d-825b-2ecf6f829512-kube-api-access-gmcjq" (OuterVolumeSpecName: "kube-api-access-gmcjq") pod "fd9fc061-0dd3-403d-825b-2ecf6f829512" (UID: "fd9fc061-0dd3-403d-825b-2ecf6f829512"). InnerVolumeSpecName "kube-api-access-gmcjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:45:40.893339 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.893251 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312aa157-e1b4-42fe-b152-401e9a0424cd-kube-api-access-dpgd9" (OuterVolumeSpecName: "kube-api-access-dpgd9") pod "312aa157-e1b4-42fe-b152-401e9a0424cd" (UID: "312aa157-e1b4-42fe-b152-401e9a0424cd"). InnerVolumeSpecName "kube-api-access-dpgd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:45:40.895071 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.895045 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-2mthm" Jul 14 21:45:40.992401 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.992360 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8vgh\" (UniqueName: \"kubernetes.io/projected/c2fec568-4024-4ae1-b11d-0c325f291066-kube-api-access-c8vgh\") pod \"c2fec568-4024-4ae1-b11d-0c325f291066\" (UID: \"c2fec568-4024-4ae1-b11d-0c325f291066\") " Jul 14 21:45:40.992583 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.992508 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpgd9\" (UniqueName: \"kubernetes.io/projected/312aa157-e1b4-42fe-b152-401e9a0424cd-kube-api-access-dpgd9\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:45:40.992583 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.992519 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmcjq\" (UniqueName: \"kubernetes.io/projected/fd9fc061-0dd3-403d-825b-2ecf6f829512-kube-api-access-gmcjq\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:45:40.994419 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:40.994394 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fec568-4024-4ae1-b11d-0c325f291066-kube-api-access-c8vgh" (OuterVolumeSpecName: "kube-api-access-c8vgh") pod "c2fec568-4024-4ae1-b11d-0c325f291066" (UID: "c2fec568-4024-4ae1-b11d-0c325f291066"). InnerVolumeSpecName "kube-api-access-c8vgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:45:41.093076 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.092979 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8vgh\" (UniqueName: \"kubernetes.io/projected/c2fec568-4024-4ae1-b11d-0c325f291066-kube-api-access-c8vgh\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:45:41.525511 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.525412 2552 generic.go:356] "Generic (PLEG): container finished" podID="c2fec568-4024-4ae1-b11d-0c325f291066" containerID="1dbee9441a03f5ca055729b63ebcdc8cea64f904cc31a16491a36d8800f33634" exitCode=0 Jul 14 21:45:41.525511 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.525455 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-2mthm" Jul 14 21:45:41.525511 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.525497 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-2mthm" event={"ID":"c2fec568-4024-4ae1-b11d-0c325f291066","Type":"ContainerDied","Data":"1dbee9441a03f5ca055729b63ebcdc8cea64f904cc31a16491a36d8800f33634"} Jul 14 21:45:41.525785 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.525537 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-2mthm" event={"ID":"c2fec568-4024-4ae1-b11d-0c325f291066","Type":"ContainerDied","Data":"3995d9f26623f562c99e581ffd253678fbf9374a415df00fdbe6dfc5cf609d73"} Jul 14 21:45:41.525785 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.525560 2552 scope.go:117] "RemoveContainer" containerID="1dbee9441a03f5ca055729b63ebcdc8cea64f904cc31a16491a36d8800f33634" Jul 14 21:45:41.526897 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.526812 2552 generic.go:356] "Generic (PLEG): container finished" podID="312aa157-e1b4-42fe-b152-401e9a0424cd" containerID="da76609f64ca81fbccbbed36e57daee3930f246ad6506d1aea0720fceb80aa2e" exitCode=0 Jul 14 21:45:41.526897 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.526862 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-865b878667-gxdjs" Jul 14 21:45:41.527023 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.526893 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-865b878667-gxdjs" event={"ID":"312aa157-e1b4-42fe-b152-401e9a0424cd","Type":"ContainerDied","Data":"da76609f64ca81fbccbbed36e57daee3930f246ad6506d1aea0720fceb80aa2e"} Jul 14 21:45:41.527023 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.526931 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-865b878667-gxdjs" event={"ID":"312aa157-e1b4-42fe-b152-401e9a0424cd","Type":"ContainerDied","Data":"2f1bf5d1a48e98eef8bf90e021493e77c94fb2795a872ba37639fbcaf2b1c294"} Jul 14 21:45:41.528453 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.528430 2552 generic.go:356] "Generic (PLEG): container finished" podID="fd9fc061-0dd3-403d-825b-2ecf6f829512" containerID="e9d4f2afb8beed7942fd340a486277dbfb1ef3a6849c678d515c5c15aea2076b" exitCode=0 Jul 14 21:45:41.528610 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.528486 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sjpj6" Jul 14 21:45:41.528610 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.528501 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sjpj6" event={"ID":"fd9fc061-0dd3-403d-825b-2ecf6f829512","Type":"ContainerDied","Data":"e9d4f2afb8beed7942fd340a486277dbfb1ef3a6849c678d515c5c15aea2076b"} Jul 14 21:45:41.528610 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.528548 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sjpj6" event={"ID":"fd9fc061-0dd3-403d-825b-2ecf6f829512","Type":"ContainerDied","Data":"43256cb10f5378850048c62c839e100427950186501b9c65773752631a12419d"} Jul 14 21:45:41.537220 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.537167 2552 scope.go:117] "RemoveContainer" containerID="1dbee9441a03f5ca055729b63ebcdc8cea64f904cc31a16491a36d8800f33634" Jul 14 21:45:41.537666 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:45:41.537638 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dbee9441a03f5ca055729b63ebcdc8cea64f904cc31a16491a36d8800f33634\": container with ID starting with 1dbee9441a03f5ca055729b63ebcdc8cea64f904cc31a16491a36d8800f33634 not found: ID does not exist" containerID="1dbee9441a03f5ca055729b63ebcdc8cea64f904cc31a16491a36d8800f33634" Jul 14 21:45:41.543227 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.538091 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbee9441a03f5ca055729b63ebcdc8cea64f904cc31a16491a36d8800f33634"} err="failed to get container status \"1dbee9441a03f5ca055729b63ebcdc8cea64f904cc31a16491a36d8800f33634\": rpc error: code = NotFound desc = could not find container \"1dbee9441a03f5ca055729b63ebcdc8cea64f904cc31a16491a36d8800f33634\": container with ID starting with 1dbee9441a03f5ca055729b63ebcdc8cea64f904cc31a16491a36d8800f33634 not found: ID does not exist" Jul 14 21:45:41.543227 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.538132 2552 scope.go:117] "RemoveContainer" containerID="da76609f64ca81fbccbbed36e57daee3930f246ad6506d1aea0720fceb80aa2e" Jul 14 21:45:41.552073 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.552055 2552 scope.go:117] "RemoveContainer" containerID="da76609f64ca81fbccbbed36e57daee3930f246ad6506d1aea0720fceb80aa2e" Jul 14 21:45:41.552348 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:45:41.552329 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da76609f64ca81fbccbbed36e57daee3930f246ad6506d1aea0720fceb80aa2e\": container with ID starting with da76609f64ca81fbccbbed36e57daee3930f246ad6506d1aea0720fceb80aa2e not found: ID does not exist" containerID="da76609f64ca81fbccbbed36e57daee3930f246ad6506d1aea0720fceb80aa2e" Jul 14 21:45:41.552446 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.552357 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da76609f64ca81fbccbbed36e57daee3930f246ad6506d1aea0720fceb80aa2e"} err="failed to get container status \"da76609f64ca81fbccbbed36e57daee3930f246ad6506d1aea0720fceb80aa2e\": rpc error: code = NotFound desc = could not find container \"da76609f64ca81fbccbbed36e57daee3930f246ad6506d1aea0720fceb80aa2e\": container with ID starting with da76609f64ca81fbccbbed36e57daee3930f246ad6506d1aea0720fceb80aa2e not found: ID does not exist" Jul 14 21:45:41.552446 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.552372 2552 scope.go:117] "RemoveContainer" containerID="e9d4f2afb8beed7942fd340a486277dbfb1ef3a6849c678d515c5c15aea2076b" Jul 14 21:45:41.559973 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.559956 2552 scope.go:117] "RemoveContainer" containerID="e9d4f2afb8beed7942fd340a486277dbfb1ef3a6849c678d515c5c15aea2076b" Jul 14 21:45:41.560247 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:45:41.560227 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d4f2afb8beed7942fd340a486277dbfb1ef3a6849c678d515c5c15aea2076b\": container with ID starting with e9d4f2afb8beed7942fd340a486277dbfb1ef3a6849c678d515c5c15aea2076b not found: ID does not exist" containerID="e9d4f2afb8beed7942fd340a486277dbfb1ef3a6849c678d515c5c15aea2076b" Jul 14 21:45:41.560336 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.560252 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d4f2afb8beed7942fd340a486277dbfb1ef3a6849c678d515c5c15aea2076b"} err="failed to get container status \"e9d4f2afb8beed7942fd340a486277dbfb1ef3a6849c678d515c5c15aea2076b\": rpc error: code = NotFound desc = could not find container \"e9d4f2afb8beed7942fd340a486277dbfb1ef3a6849c678d515c5c15aea2076b\": container with ID starting with e9d4f2afb8beed7942fd340a486277dbfb1ef3a6849c678d515c5c15aea2076b not found: ID does not exist" Jul 14 21:45:41.566328 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.566307 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-865b878667-gxdjs"] Jul 14 21:45:41.578477 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.578455 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-865b878667-gxdjs"] Jul 14 21:45:41.594092 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.594061 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2mthm"] Jul 14 21:45:41.602657 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.602631 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2mthm"] Jul 14 21:45:41.618028 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.618002 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjpj6"] Jul 14 21:45:41.626765 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.626745 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjpj6"] Jul 14 21:45:41.992163 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.992133 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="312aa157-e1b4-42fe-b152-401e9a0424cd" path="/var/lib/kubelet/pods/312aa157-e1b4-42fe-b152-401e9a0424cd/volumes" Jul 14 21:45:41.992570 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.992523 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2fec568-4024-4ae1-b11d-0c325f291066" path="/var/lib/kubelet/pods/c2fec568-4024-4ae1-b11d-0c325f291066/volumes" Jul 14 21:45:41.992853 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:41.992839 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd9fc061-0dd3-403d-825b-2ecf6f829512" path="/var/lib/kubelet/pods/fd9fc061-0dd3-403d-825b-2ecf6f829512/volumes" Jul 14 21:45:45.158975 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.158941 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:45.172994 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.172967 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7956fbf6dc-tx75j"] Jul 14 21:45:45.173305 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.173293 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2fec568-4024-4ae1-b11d-0c325f291066" containerName="authorino" Jul 14 21:45:45.173354 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.173307 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2fec568-4024-4ae1-b11d-0c325f291066" containerName="authorino" Jul 14 21:45:45.173354 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.173321 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd9fc061-0dd3-403d-825b-2ecf6f829512" containerName="authorino" Jul 14 21:45:45.173354 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.173326 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd9fc061-0dd3-403d-825b-2ecf6f829512" containerName="authorino" Jul 14 21:45:45.173354 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.173340 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="312aa157-e1b4-42fe-b152-401e9a0424cd" containerName="authorino" Jul 14 21:45:45.173354 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.173345 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="312aa157-e1b4-42fe-b152-401e9a0424cd" containerName="authorino" Jul 14 21:45:45.173504 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.173404 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2fec568-4024-4ae1-b11d-0c325f291066" containerName="authorino" Jul 14 21:45:45.173504 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.173411 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="312aa157-e1b4-42fe-b152-401e9a0424cd" containerName="authorino" Jul 14 21:45:45.173504 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.173419 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd9fc061-0dd3-403d-825b-2ecf6f829512" containerName="authorino" Jul 14 21:45:45.177811 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.177792 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7956fbf6dc-tx75j" Jul 14 21:45:45.180597 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.180575 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9dr4p\"" Jul 14 21:45:45.180717 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.180616 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 14 21:45:45.180717 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.180616 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 14 21:45:45.188599 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.188572 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7956fbf6dc-tx75j"] Jul 14 21:45:45.229804 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.229766 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c309a270-3ecd-40cd-a556-def9a04d03f9-webhook-cert\") pod \"maas-controller-7956fbf6dc-tx75j\" (UID: \"c309a270-3ecd-40cd-a556-def9a04d03f9\") " pod="opendatahub/maas-controller-7956fbf6dc-tx75j" Jul 14 21:45:45.229952 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.229827 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7qwh\" (UniqueName: \"kubernetes.io/projected/c309a270-3ecd-40cd-a556-def9a04d03f9-kube-api-access-n7qwh\") pod \"maas-controller-7956fbf6dc-tx75j\" (UID: \"c309a270-3ecd-40cd-a556-def9a04d03f9\") " pod="opendatahub/maas-controller-7956fbf6dc-tx75j" Jul 14 21:45:45.261140 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.261103 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:45.330700 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.330662 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c309a270-3ecd-40cd-a556-def9a04d03f9-webhook-cert\") pod \"maas-controller-7956fbf6dc-tx75j\" (UID: \"c309a270-3ecd-40cd-a556-def9a04d03f9\") " pod="opendatahub/maas-controller-7956fbf6dc-tx75j" Jul 14 21:45:45.330855 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.330756 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7qwh\" (UniqueName: \"kubernetes.io/projected/c309a270-3ecd-40cd-a556-def9a04d03f9-kube-api-access-n7qwh\") pod \"maas-controller-7956fbf6dc-tx75j\" (UID: \"c309a270-3ecd-40cd-a556-def9a04d03f9\") " pod="opendatahub/maas-controller-7956fbf6dc-tx75j" Jul 14 21:45:45.330855 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:45:45.330823 2552 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 14 21:45:45.330931 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:45:45.330915 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c309a270-3ecd-40cd-a556-def9a04d03f9-webhook-cert podName:c309a270-3ecd-40cd-a556-def9a04d03f9 nodeName:}" failed. No retries permitted until 2026-07-14 21:45:45.830895828 +0000 UTC m=+512.369601489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/c309a270-3ecd-40cd-a556-def9a04d03f9-webhook-cert") pod "maas-controller-7956fbf6dc-tx75j" (UID: "c309a270-3ecd-40cd-a556-def9a04d03f9") : secret "maas-controller-webhook-cert" not found Jul 14 21:45:45.345794 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.345757 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7qwh\" (UniqueName: \"kubernetes.io/projected/c309a270-3ecd-40cd-a556-def9a04d03f9-kube-api-access-n7qwh\") pod \"maas-controller-7956fbf6dc-tx75j\" (UID: \"c309a270-3ecd-40cd-a556-def9a04d03f9\") " pod="opendatahub/maas-controller-7956fbf6dc-tx75j" Jul 14 21:45:45.347369 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.347342 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:45.629538 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.629504 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:45.835775 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.835739 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c309a270-3ecd-40cd-a556-def9a04d03f9-webhook-cert\") pod \"maas-controller-7956fbf6dc-tx75j\" (UID: \"c309a270-3ecd-40cd-a556-def9a04d03f9\") " pod="opendatahub/maas-controller-7956fbf6dc-tx75j" Jul 14 21:45:45.838535 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:45.838469 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c309a270-3ecd-40cd-a556-def9a04d03f9-webhook-cert\") pod \"maas-controller-7956fbf6dc-tx75j\" (UID: \"c309a270-3ecd-40cd-a556-def9a04d03f9\") " pod="opendatahub/maas-controller-7956fbf6dc-tx75j" Jul 14 21:45:46.089628 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:46.089595 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7956fbf6dc-tx75j" Jul 14 21:45:46.232149 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:46.232124 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7956fbf6dc-tx75j"] Jul 14 21:45:46.234485 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:45:46.234452 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc309a270_3ecd_40cd_a556_def9a04d03f9.slice/crio-f5f1f03c54e68f285e8f8656a420391bcc3486d0d4063a63570d919c3efde463 WatchSource:0}: Error finding container f5f1f03c54e68f285e8f8656a420391bcc3486d0d4063a63570d919c3efde463: Status 404 returned error can't find the container with id f5f1f03c54e68f285e8f8656a420391bcc3486d0d4063a63570d919c3efde463 Jul 14 21:45:46.559225 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:46.559119 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7956fbf6dc-tx75j" event={"ID":"c309a270-3ecd-40cd-a556-def9a04d03f9","Type":"ContainerStarted","Data":"f5f1f03c54e68f285e8f8656a420391bcc3486d0d4063a63570d919c3efde463"} Jul 14 21:45:48.527679 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:48.527536 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:48.575215 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:48.575155 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7956fbf6dc-tx75j" event={"ID":"c309a270-3ecd-40cd-a556-def9a04d03f9","Type":"ContainerStarted","Data":"26495af77c373fc5582bfde4f4a28509d12ec04af7676ec804da8fc205832bbc"} Jul 14 21:45:48.575387 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:48.575274 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7956fbf6dc-tx75j" Jul 14 21:45:48.600179 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:48.599964 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7956fbf6dc-tx75j" podStartSLOduration=1.490082194 podStartE2EDuration="3.599943946s" totalImagesPullingTime="2.109861752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:45:48.598907067 +0000 UTC m=+515.137612750" watchObservedRunningTime="2026-07-14 21:45:48.599943946 +0000 UTC m=+515.138649619" Jul 14 21:45:48.611964 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:48.611922 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:45:59.589217 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:45:59.589160 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7956fbf6dc-tx75j" Jul 14 21:46:00.367434 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.367376 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7956fbf6dc-tx75j"] Jul 14 21:46:00.367804 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.367748 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7956fbf6dc-tx75j" podUID="c309a270-3ecd-40cd-a556-def9a04d03f9" containerName="manager" containerID="cri-o://26495af77c373fc5582bfde4f4a28509d12ec04af7676ec804da8fc205832bbc" gracePeriod=30 Jul 14 21:46:00.615877 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.615851 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7956fbf6dc-tx75j" Jul 14 21:46:00.646367 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.646281 2552 generic.go:356] "Generic (PLEG): container finished" podID="c309a270-3ecd-40cd-a556-def9a04d03f9" containerID="26495af77c373fc5582bfde4f4a28509d12ec04af7676ec804da8fc205832bbc" exitCode=0 Jul 14 21:46:00.646367 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.646320 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7956fbf6dc-tx75j" Jul 14 21:46:00.646545 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.646364 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7956fbf6dc-tx75j" event={"ID":"c309a270-3ecd-40cd-a556-def9a04d03f9","Type":"ContainerDied","Data":"26495af77c373fc5582bfde4f4a28509d12ec04af7676ec804da8fc205832bbc"} Jul 14 21:46:00.646545 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.646415 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7956fbf6dc-tx75j" event={"ID":"c309a270-3ecd-40cd-a556-def9a04d03f9","Type":"ContainerDied","Data":"f5f1f03c54e68f285e8f8656a420391bcc3486d0d4063a63570d919c3efde463"} Jul 14 21:46:00.646545 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.646438 2552 scope.go:117] "RemoveContainer" containerID="26495af77c373fc5582bfde4f4a28509d12ec04af7676ec804da8fc205832bbc" Jul 14 21:46:00.655521 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.655502 2552 scope.go:117] "RemoveContainer" containerID="26495af77c373fc5582bfde4f4a28509d12ec04af7676ec804da8fc205832bbc" Jul 14 21:46:00.655814 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:46:00.655797 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26495af77c373fc5582bfde4f4a28509d12ec04af7676ec804da8fc205832bbc\": container with ID starting with 26495af77c373fc5582bfde4f4a28509d12ec04af7676ec804da8fc205832bbc not found: ID does not exist" containerID="26495af77c373fc5582bfde4f4a28509d12ec04af7676ec804da8fc205832bbc" Jul 14 21:46:00.655861 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.655826 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26495af77c373fc5582bfde4f4a28509d12ec04af7676ec804da8fc205832bbc"} err="failed to get container status \"26495af77c373fc5582bfde4f4a28509d12ec04af7676ec804da8fc205832bbc\": rpc error: code = NotFound desc = could not find container \"26495af77c373fc5582bfde4f4a28509d12ec04af7676ec804da8fc205832bbc\": container with ID starting with 26495af77c373fc5582bfde4f4a28509d12ec04af7676ec804da8fc205832bbc not found: ID does not exist" Jul 14 21:46:00.668621 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.668591 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c309a270-3ecd-40cd-a556-def9a04d03f9-webhook-cert\") pod \"c309a270-3ecd-40cd-a556-def9a04d03f9\" (UID: \"c309a270-3ecd-40cd-a556-def9a04d03f9\") " Jul 14 21:46:00.668773 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.668653 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7qwh\" (UniqueName: \"kubernetes.io/projected/c309a270-3ecd-40cd-a556-def9a04d03f9-kube-api-access-n7qwh\") pod \"c309a270-3ecd-40cd-a556-def9a04d03f9\" (UID: \"c309a270-3ecd-40cd-a556-def9a04d03f9\") " Jul 14 21:46:00.670894 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.670860 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c309a270-3ecd-40cd-a556-def9a04d03f9-kube-api-access-n7qwh" (OuterVolumeSpecName: "kube-api-access-n7qwh") pod "c309a270-3ecd-40cd-a556-def9a04d03f9" (UID: "c309a270-3ecd-40cd-a556-def9a04d03f9"). InnerVolumeSpecName "kube-api-access-n7qwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:46:00.670894 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.670872 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c309a270-3ecd-40cd-a556-def9a04d03f9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c309a270-3ecd-40cd-a556-def9a04d03f9" (UID: "c309a270-3ecd-40cd-a556-def9a04d03f9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:46:00.770269 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.770186 2552 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c309a270-3ecd-40cd-a556-def9a04d03f9-webhook-cert\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:46:00.770269 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.770263 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7qwh\" (UniqueName: \"kubernetes.io/projected/c309a270-3ecd-40cd-a556-def9a04d03f9-kube-api-access-n7qwh\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:46:00.972808 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.972769 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7956fbf6dc-tx75j"] Jul 14 21:46:00.975488 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:00.975460 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7956fbf6dc-tx75j"] Jul 14 21:46:01.314818 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.314726 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-664d86cc8-wmwlp" podUID="2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f" containerName="console" containerID="cri-o://b448864302d20e23d087f35ab1ae91c7ca7882756c670f13879667d08d1baba5" gracePeriod=15 Jul 14 21:46:01.577241 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.577166 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:46:01.652908 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.652875 2552 generic.go:356] "Generic (PLEG): container finished" podID="2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f" containerID="b448864302d20e23d087f35ab1ae91c7ca7882756c670f13879667d08d1baba5" exitCode=0 Jul 14 21:46:01.653368 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.652921 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664d86cc8-wmwlp" Jul 14 21:46:01.653368 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.652958 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664d86cc8-wmwlp" event={"ID":"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f","Type":"ContainerDied","Data":"b448864302d20e23d087f35ab1ae91c7ca7882756c670f13879667d08d1baba5"} Jul 14 21:46:01.653368 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.653008 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664d86cc8-wmwlp" event={"ID":"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f","Type":"ContainerDied","Data":"c71f6052d051787ff200d66168108835f7c9eb1aac1f75e2a4360065f4c446e7"} Jul 14 21:46:01.653368 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.653035 2552 scope.go:117] "RemoveContainer" containerID="b448864302d20e23d087f35ab1ae91c7ca7882756c670f13879667d08d1baba5" Jul 14 21:46:01.661823 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.661792 2552 scope.go:117] "RemoveContainer" containerID="b448864302d20e23d087f35ab1ae91c7ca7882756c670f13879667d08d1baba5" Jul 14 21:46:01.662077 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:46:01.662057 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b448864302d20e23d087f35ab1ae91c7ca7882756c670f13879667d08d1baba5\": container with ID starting with b448864302d20e23d087f35ab1ae91c7ca7882756c670f13879667d08d1baba5 not found: ID does not exist" containerID="b448864302d20e23d087f35ab1ae91c7ca7882756c670f13879667d08d1baba5" Jul 14 21:46:01.662151 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.662091 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b448864302d20e23d087f35ab1ae91c7ca7882756c670f13879667d08d1baba5"} err="failed to get container status \"b448864302d20e23d087f35ab1ae91c7ca7882756c670f13879667d08d1baba5\": rpc error: code = NotFound desc = could not find container \"b448864302d20e23d087f35ab1ae91c7ca7882756c670f13879667d08d1baba5\": container with ID starting with b448864302d20e23d087f35ab1ae91c7ca7882756c670f13879667d08d1baba5 not found: ID does not exist" Jul 14 21:46:01.678361 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.678336 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-trusted-ca-bundle\") pod \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " Jul 14 21:46:01.678474 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.678376 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-oauth-config\") pod \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " Jul 14 21:46:01.678474 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.678395 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6br4t\" (UniqueName: \"kubernetes.io/projected/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-kube-api-access-6br4t\") pod \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " Jul 14 21:46:01.678474 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.678447 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-oauth-serving-cert\") pod \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " Jul 14 21:46:01.678474 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.678461 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-config\") pod \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " Jul 14 21:46:01.678669 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.678477 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-serving-cert\") pod \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " Jul 14 21:46:01.678669 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.678496 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-service-ca\") pod \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\" (UID: \"2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f\") " Jul 14 21:46:01.679034 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.678919 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f" (UID: "2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:46:01.679034 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.678954 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-config" (OuterVolumeSpecName: "console-config") pod "2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f" (UID: "2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:46:01.679034 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.678979 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-service-ca" (OuterVolumeSpecName: "service-ca") pod "2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f" (UID: "2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:46:01.679034 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.678996 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f" (UID: "2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 21:46:01.680626 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.680598 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f" (UID: "2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:46:01.680731 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.680646 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f" (UID: "2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:46:01.680731 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.680716 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-kube-api-access-6br4t" (OuterVolumeSpecName: "kube-api-access-6br4t") pod "2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f" (UID: "2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f"). InnerVolumeSpecName "kube-api-access-6br4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:46:01.779634 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.779593 2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-oauth-serving-cert\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:46:01.779634 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.779627 2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-config\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:46:01.779634 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.779637 2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-serving-cert\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:46:01.779634 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.779647 2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:46:01.779915 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.779656 2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-trusted-ca-bundle\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:46:01.779915 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.779664 2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-console-oauth-config\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:46:01.779915 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.779672 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6br4t\" (UniqueName: \"kubernetes.io/projected/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f-kube-api-access-6br4t\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:46:01.979650 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.979619 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-664d86cc8-wmwlp"] Jul 14 21:46:01.984085 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.984057 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-664d86cc8-wmwlp"] Jul 14 21:46:01.991485 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.991457 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f" path="/var/lib/kubelet/pods/2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f/volumes" Jul 14 21:46:01.991825 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:01.991812 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c309a270-3ecd-40cd-a556-def9a04d03f9" path="/var/lib/kubelet/pods/c309a270-3ecd-40cd-a556-def9a04d03f9/volumes" Jul 14 21:46:19.942058 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:19.942012 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:46:20.359126 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:20.359095 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:46:20.728896 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:20.728796 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:46:20.971081 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:20.971046 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:46:21.080634 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.080530 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:46:21.115118 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.115073 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-zkj4p"] Jul 14 21:46:21.115777 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.115754 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f" containerName="console" Jul 14 21:46:21.115777 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.115778 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f" containerName="console" Jul 14 21:46:21.115966 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.115802 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c309a270-3ecd-40cd-a556-def9a04d03f9" containerName="manager" Jul 14 21:46:21.115966 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.115811 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c309a270-3ecd-40cd-a556-def9a04d03f9" containerName="manager" Jul 14 21:46:21.116061 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.115996 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9ecf68-daeb-4bb4-8280-247e6e8b0e4f" containerName="console" Jul 14 21:46:21.116061 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.116022 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c309a270-3ecd-40cd-a556-def9a04d03f9" containerName="manager" Jul 14 21:46:21.121095 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.121070 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-zkj4p" Jul 14 21:46:21.124998 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.124974 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vbhsp\"" Jul 14 21:46:21.125422 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.125405 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 14 21:46:21.132148 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.132125 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-zkj4p"] Jul 14 21:46:21.251416 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.251380 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75m8t\" (UniqueName: \"kubernetes.io/projected/14003eef-8c35-409d-b56d-62d729ad89d0-kube-api-access-75m8t\") pod \"payload-processing-58f86c7798-zkj4p\" (UID: \"14003eef-8c35-409d-b56d-62d729ad89d0\") " pod="openshift-ingress/payload-processing-58f86c7798-zkj4p" Jul 14 21:46:21.251613 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.251453 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14003eef-8c35-409d-b56d-62d729ad89d0-plugins-config-volume\") pod \"payload-processing-58f86c7798-zkj4p\" (UID: \"14003eef-8c35-409d-b56d-62d729ad89d0\") " pod="openshift-ingress/payload-processing-58f86c7798-zkj4p" Jul 14 21:46:21.352124 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.352090 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14003eef-8c35-409d-b56d-62d729ad89d0-plugins-config-volume\") pod \"payload-processing-58f86c7798-zkj4p\" (UID: \"14003eef-8c35-409d-b56d-62d729ad89d0\") " pod="openshift-ingress/payload-processing-58f86c7798-zkj4p" Jul 14 21:46:21.352333 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.352146 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75m8t\" (UniqueName: \"kubernetes.io/projected/14003eef-8c35-409d-b56d-62d729ad89d0-kube-api-access-75m8t\") pod \"payload-processing-58f86c7798-zkj4p\" (UID: \"14003eef-8c35-409d-b56d-62d729ad89d0\") " pod="openshift-ingress/payload-processing-58f86c7798-zkj4p" Jul 14 21:46:21.352870 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.352846 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14003eef-8c35-409d-b56d-62d729ad89d0-plugins-config-volume\") pod \"payload-processing-58f86c7798-zkj4p\" (UID: \"14003eef-8c35-409d-b56d-62d729ad89d0\") " pod="openshift-ingress/payload-processing-58f86c7798-zkj4p" Jul 14 21:46:21.367140 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.367107 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75m8t\" (UniqueName: \"kubernetes.io/projected/14003eef-8c35-409d-b56d-62d729ad89d0-kube-api-access-75m8t\") pod \"payload-processing-58f86c7798-zkj4p\" (UID: \"14003eef-8c35-409d-b56d-62d729ad89d0\") " pod="openshift-ingress/payload-processing-58f86c7798-zkj4p" Jul 14 21:46:21.432001 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.431967 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-zkj4p" Jul 14 21:46:21.575945 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.575920 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-zkj4p"] Jul 14 21:46:21.577496 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:46:21.577467 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14003eef_8c35_409d_b56d_62d729ad89d0.slice/crio-ee9dd066f2801e176098fd4f085f35705710bb7adf4b6ea43a0a5d50ac451aa8 WatchSource:0}: Error finding container ee9dd066f2801e176098fd4f085f35705710bb7adf4b6ea43a0a5d50ac451aa8: Status 404 returned error can't find the container with id ee9dd066f2801e176098fd4f085f35705710bb7adf4b6ea43a0a5d50ac451aa8 Jul 14 21:46:21.759737 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:21.759654 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-zkj4p" event={"ID":"14003eef-8c35-409d-b56d-62d729ad89d0","Type":"ContainerStarted","Data":"ee9dd066f2801e176098fd4f085f35705710bb7adf4b6ea43a0a5d50ac451aa8"} Jul 14 21:46:23.771814 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:23.771777 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-zkj4p" event={"ID":"14003eef-8c35-409d-b56d-62d729ad89d0","Type":"ContainerStarted","Data":"c874922e0762e521ad0fc5d11d3081af75ce6593186e72e3c08df203df744c0a"} Jul 14 21:46:23.772317 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:23.771844 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-zkj4p" Jul 14 21:46:23.791385 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:23.791305 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-zkj4p" podStartSLOduration=1.479373517 podStartE2EDuration="2.791285354s" totalImagesPullingTime="1.311911837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:46:23.790918693 +0000 UTC m=+550.329624366" watchObservedRunningTime="2026-07-14 21:46:23.791285354 +0000 UTC m=+550.329991027" Jul 14 21:46:30.866421 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:30.866390 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:46:30.977736 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:30.977699 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:46:34.779111 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:34.779081 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-zkj4p" Jul 14 21:46:36.972470 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:36.972437 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:46:37.646744 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:37.646709 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:46:37.752505 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:37.752462 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:46:47.046395 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:47.046359 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:46:48.392628 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.392593 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn"] Jul 14 21:46:48.395905 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.395882 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.400529 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.400472 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 14 21:46:48.400529 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.400518 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gch6t\"" Jul 14 21:46:48.400734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.400506 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 14 21:46:48.400734 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.400502 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 14 21:46:48.407637 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.407612 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn"] Jul 14 21:46:48.418823 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.418791 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:46:48.454772 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.454739 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.454772 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.454785 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.454985 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.454854 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.454985 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.454875 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.454985 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.454907 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb6g5\" (UniqueName: \"kubernetes.io/projected/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-kube-api-access-tb6g5\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.454985 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.454938 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.556213 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.556149 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.556423 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.556224 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.556423 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.556261 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb6g5\" (UniqueName: \"kubernetes.io/projected/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-kube-api-access-tb6g5\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.556423 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.556408 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.556616 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.556470 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.556616 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.556532 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.556616 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.556582 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.556779 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.556761 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.556872 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.556853 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.558740 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.558711 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.558950 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.558930 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.568407 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.568383 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb6g5\" (UniqueName: \"kubernetes.io/projected/7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e-kube-api-access-tb6g5\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ngcmn\" (UID: \"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.706824 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.706725 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:48.861958 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.861927 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn"] Jul 14 21:46:48.865553 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:46:48.865518 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af7a3a1_bcfb_4ceb_b66a_ef97b4aa976e.slice/crio-aab1d26bd5fc78ad2d54728b23549a919e8de4bbb0972c31530dda1165b73f2c WatchSource:0}: Error finding container aab1d26bd5fc78ad2d54728b23549a919e8de4bbb0972c31530dda1165b73f2c: Status 404 returned error can't find the container with id aab1d26bd5fc78ad2d54728b23549a919e8de4bbb0972c31530dda1165b73f2c Jul 14 21:46:48.905243 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:48.905183 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" event={"ID":"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e","Type":"ContainerStarted","Data":"aab1d26bd5fc78ad2d54728b23549a919e8de4bbb0972c31530dda1165b73f2c"} Jul 14 21:46:50.919861 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:50.919823 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" event={"ID":"7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e","Type":"ContainerStarted","Data":"e2a470041e2b1c292ecd2e022fafd171f4db1e27622cf788a5168409b92f089a"} Jul 14 21:46:50.941674 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:50.941617 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" podStartSLOduration=1.28413489 podStartE2EDuration="2.941596168s" totalImagesPullingTime="1.657461278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:46:50.939727189 +0000 UTC m=+577.478432871" watchObservedRunningTime="2026-07-14 21:46:50.941596168 +0000 UTC m=+577.480301840" Jul 14 21:46:51.404999 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:51.404964 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:46:58.707120 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:58.707085 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:58.707612 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:58.707168 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:58.719811 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:58.719783 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:46:58.970561 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:46:58.970479 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ngcmn" Jul 14 21:47:02.819665 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:02.819626 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:47:07.623245 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:07.623213 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:47:13.939245 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:13.939211 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 21:47:13.939702 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:13.939636 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 21:47:13.945917 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:13.945895 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 21:47:13.946176 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:13.946129 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 21:47:17.472985 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.472951 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh"] Jul 14 21:47:17.476515 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.476496 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.479771 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.479743 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 14 21:47:17.494286 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.494260 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh"] Jul 14 21:47:17.523248 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.523217 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:47:17.613379 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.613342 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12db4e4-83ee-4771-a746-94449786b2ea-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.613589 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.613397 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12db4e4-83ee-4771-a746-94449786b2ea-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.613589 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.613420 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12db4e4-83ee-4771-a746-94449786b2ea-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.613589 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.613442 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78zxl\" (UniqueName: \"kubernetes.io/projected/c12db4e4-83ee-4771-a746-94449786b2ea-kube-api-access-78zxl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.613589 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.613463 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12db4e4-83ee-4771-a746-94449786b2ea-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.613589 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.613484 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12db4e4-83ee-4771-a746-94449786b2ea-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.714040 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.713997 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12db4e4-83ee-4771-a746-94449786b2ea-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.714040 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.714041 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12db4e4-83ee-4771-a746-94449786b2ea-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.714350 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.714072 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78zxl\" (UniqueName: \"kubernetes.io/projected/c12db4e4-83ee-4771-a746-94449786b2ea-kube-api-access-78zxl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.714350 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.714105 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12db4e4-83ee-4771-a746-94449786b2ea-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.714350 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.714137 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12db4e4-83ee-4771-a746-94449786b2ea-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.714350 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.714237 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12db4e4-83ee-4771-a746-94449786b2ea-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.714573 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.714550 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12db4e4-83ee-4771-a746-94449786b2ea-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.714627 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.714560 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12db4e4-83ee-4771-a746-94449786b2ea-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.714665 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.714642 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12db4e4-83ee-4771-a746-94449786b2ea-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.716557 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.716534 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12db4e4-83ee-4771-a746-94449786b2ea-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.716810 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.716794 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12db4e4-83ee-4771-a746-94449786b2ea-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.724866 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.724803 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78zxl\" (UniqueName: \"kubernetes.io/projected/c12db4e4-83ee-4771-a746-94449786b2ea-kube-api-access-78zxl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zpmnh\" (UID: \"c12db4e4-83ee-4771-a746-94449786b2ea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.787513 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.787474 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:17.925511 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:17.925481 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh"] Jul 14 21:47:17.927105 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:47:17.927080 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12db4e4_83ee_4771_a746_94449786b2ea.slice/crio-e2bb40d011205e927fdc18edc558ee4f09aafbc9f84a7103dd5c52815b54770c WatchSource:0}: Error finding container e2bb40d011205e927fdc18edc558ee4f09aafbc9f84a7103dd5c52815b54770c: Status 404 returned error can't find the container with id e2bb40d011205e927fdc18edc558ee4f09aafbc9f84a7103dd5c52815b54770c Jul 14 21:47:18.057375 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:18.057285 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" event={"ID":"c12db4e4-83ee-4771-a746-94449786b2ea","Type":"ContainerStarted","Data":"e2bb40d011205e927fdc18edc558ee4f09aafbc9f84a7103dd5c52815b54770c"} Jul 14 21:47:19.064025 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:19.063989 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" event={"ID":"c12db4e4-83ee-4771-a746-94449786b2ea","Type":"ContainerStarted","Data":"97520d4dbd0726a39a4917504dbbc40b6a44c5c2f0dc9b6c2bec141fad1d5cfc"} Jul 14 21:47:19.084392 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:19.084345 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" podStartSLOduration=1.862772656 podStartE2EDuration="2.084329602s" totalImagesPullingTime="221.556946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:47:19.084300361 +0000 UTC m=+605.623006034" watchObservedRunningTime="2026-07-14 21:47:19.084329602 +0000 UTC m=+605.623035273" Jul 14 21:47:24.781355 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.781316 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv"] Jul 14 21:47:24.786938 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.786913 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.789559 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.789537 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 14 21:47:24.796292 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.796267 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv"] Jul 14 21:47:24.806903 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.806876 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:47:24.878219 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.878143 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.878418 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.878280 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.878418 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.878320 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.878418 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.878366 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.878418 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.878405 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7sb7\" (UniqueName: \"kubernetes.io/projected/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-kube-api-access-t7sb7\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.878618 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.878482 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.979296 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.979255 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.979296 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.979301 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.979537 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.979429 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.979537 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.979466 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7sb7\" (UniqueName: \"kubernetes.io/projected/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-kube-api-access-t7sb7\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.979537 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.979503 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.979707 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.979557 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.979843 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.979810 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.979939 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.979846 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.979981 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.979950 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.981603 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.981581 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.981893 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.981876 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:24.988976 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:24.988950 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7sb7\" (UniqueName: \"kubernetes.io/projected/a8eeedcb-c169-4765-9ea5-ae70eb757fa9-kube-api-access-t7sb7\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv\" (UID: \"a8eeedcb-c169-4765-9ea5-ae70eb757fa9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:25.098263 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:25.098212 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:25.234890 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:25.234859 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv"] Jul 14 21:47:25.236744 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:47:25.236716 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8eeedcb_c169_4765_9ea5_ae70eb757fa9.slice/crio-d9e1d605a0c5cec15768fa894926af3f136c2a3578b4aef63e6a0dfe6c957f01 WatchSource:0}: Error finding container d9e1d605a0c5cec15768fa894926af3f136c2a3578b4aef63e6a0dfe6c957f01: Status 404 returned error can't find the container with id d9e1d605a0c5cec15768fa894926af3f136c2a3578b4aef63e6a0dfe6c957f01 Jul 14 21:47:25.238056 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:25.238037 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 21:47:26.101356 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:26.101320 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" event={"ID":"a8eeedcb-c169-4765-9ea5-ae70eb757fa9","Type":"ContainerStarted","Data":"481307819c49b9d8a9e8cbc6135817580e9b64280f5694fa6347ebf3e80bcb21"} Jul 14 21:47:26.101356 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:26.101358 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" event={"ID":"a8eeedcb-c169-4765-9ea5-ae70eb757fa9","Type":"ContainerStarted","Data":"d9e1d605a0c5cec15768fa894926af3f136c2a3578b4aef63e6a0dfe6c957f01"} Jul 14 21:47:26.125703 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:26.125650 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" podStartSLOduration=1.892978602 podStartE2EDuration="2.125633186s" totalImagesPullingTime="232.654584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:47:26.123584269 +0000 UTC m=+612.662289941" watchObservedRunningTime="2026-07-14 21:47:26.125633186 +0000 UTC m=+612.664338858" Jul 14 21:47:27.788474 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:27.788436 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:27.788885 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:27.788509 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:27.801637 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:27.801604 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:28.123173 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:28.123134 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zpmnh" Jul 14 21:47:35.098867 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:35.098824 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:35.098867 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:35.098870 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:35.111425 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:35.111401 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:47:35.158130 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:47:35.158102 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv" Jul 14 21:48:14.752673 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:14.752628 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7758664dd6-hzmr5"] Jul 14 21:48:14.755913 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:14.755895 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7758664dd6-hzmr5" Jul 14 21:48:14.765307 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:14.765275 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7758664dd6-hzmr5"] Jul 14 21:48:14.919127 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:14.919075 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s58r4\" (UniqueName: \"kubernetes.io/projected/b2cf3e3a-7a0e-4b7d-8fa9-3a485497aa21-kube-api-access-s58r4\") pod \"authorino-7758664dd6-hzmr5\" (UID: \"b2cf3e3a-7a0e-4b7d-8fa9-3a485497aa21\") " pod="kuadrant-system/authorino-7758664dd6-hzmr5" Jul 14 21:48:14.919334 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:14.919175 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b2cf3e3a-7a0e-4b7d-8fa9-3a485497aa21-tls-cert\") pod \"authorino-7758664dd6-hzmr5\" (UID: \"b2cf3e3a-7a0e-4b7d-8fa9-3a485497aa21\") " pod="kuadrant-system/authorino-7758664dd6-hzmr5" Jul 14 21:48:15.020595 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:15.020507 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s58r4\" (UniqueName: \"kubernetes.io/projected/b2cf3e3a-7a0e-4b7d-8fa9-3a485497aa21-kube-api-access-s58r4\") pod \"authorino-7758664dd6-hzmr5\" (UID: \"b2cf3e3a-7a0e-4b7d-8fa9-3a485497aa21\") " pod="kuadrant-system/authorino-7758664dd6-hzmr5" Jul 14 21:48:15.020595 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:15.020589 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b2cf3e3a-7a0e-4b7d-8fa9-3a485497aa21-tls-cert\") pod \"authorino-7758664dd6-hzmr5\" (UID: \"b2cf3e3a-7a0e-4b7d-8fa9-3a485497aa21\") " pod="kuadrant-system/authorino-7758664dd6-hzmr5" Jul 14 21:48:15.023074 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:15.023040 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b2cf3e3a-7a0e-4b7d-8fa9-3a485497aa21-tls-cert\") pod \"authorino-7758664dd6-hzmr5\" (UID: \"b2cf3e3a-7a0e-4b7d-8fa9-3a485497aa21\") " pod="kuadrant-system/authorino-7758664dd6-hzmr5" Jul 14 21:48:15.029127 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:15.029102 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s58r4\" (UniqueName: \"kubernetes.io/projected/b2cf3e3a-7a0e-4b7d-8fa9-3a485497aa21-kube-api-access-s58r4\") pod \"authorino-7758664dd6-hzmr5\" (UID: \"b2cf3e3a-7a0e-4b7d-8fa9-3a485497aa21\") " pod="kuadrant-system/authorino-7758664dd6-hzmr5" Jul 14 21:48:15.066043 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:15.066012 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7758664dd6-hzmr5" Jul 14 21:48:15.142648 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:15.142617 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:48:15.209860 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:15.209834 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7758664dd6-hzmr5"] Jul 14 21:48:15.211806 ip-10-0-130-223 kubenswrapper[2552]: W0714 21:48:15.211777 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2cf3e3a_7a0e_4b7d_8fa9_3a485497aa21.slice/crio-07d97b3afdef40858e89743243867493b4c9918a60f8d58bad560baa8644873c WatchSource:0}: Error finding container 07d97b3afdef40858e89743243867493b4c9918a60f8d58bad560baa8644873c: Status 404 returned error can't find the container with id 07d97b3afdef40858e89743243867493b4c9918a60f8d58bad560baa8644873c Jul 14 21:48:15.349352 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:15.349316 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7758664dd6-hzmr5" event={"ID":"b2cf3e3a-7a0e-4b7d-8fa9-3a485497aa21","Type":"ContainerStarted","Data":"07d97b3afdef40858e89743243867493b4c9918a60f8d58bad560baa8644873c"} Jul 14 21:48:16.356546 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:16.356505 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7758664dd6-hzmr5" event={"ID":"b2cf3e3a-7a0e-4b7d-8fa9-3a485497aa21","Type":"ContainerStarted","Data":"d4a9d5ce841bf5fe848e2d4613a1aacb37b995dc2df9ca945b93dd18ae0a0b7e"} Jul 14 21:48:16.377911 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:16.377841 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7758664dd6-hzmr5" podStartSLOduration=1.914656079 podStartE2EDuration="2.377820025s" totalImagesPullingTime="463.163946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 21:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 21:48:16.375902275 +0000 UTC m=+662.914607948" watchObservedRunningTime="2026-07-14 21:48:16.377820025 +0000 UTC m=+662.916525695" Jul 14 21:48:16.421212 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:16.420739 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c7d9f96f9-kts62"] Jul 14 21:48:16.421212 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:16.420959 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c7d9f96f9-kts62" podUID="d29997d2-a75c-4d10-a812-57d36aa57232" containerName="authorino" containerID="cri-o://65883e917af3f478f7648498590bc127626adb7352b025917c9fb2be238f2330" gracePeriod=30 Jul 14 21:48:16.702925 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:16.702887 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c7d9f96f9-kts62" Jul 14 21:48:16.727891 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:16.727851 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:48:16.839755 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:16.839717 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7mmw\" (UniqueName: \"kubernetes.io/projected/d29997d2-a75c-4d10-a812-57d36aa57232-kube-api-access-r7mmw\") pod \"d29997d2-a75c-4d10-a812-57d36aa57232\" (UID: \"d29997d2-a75c-4d10-a812-57d36aa57232\") " Jul 14 21:48:16.839964 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:16.839779 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d29997d2-a75c-4d10-a812-57d36aa57232-tls-cert\") pod \"d29997d2-a75c-4d10-a812-57d36aa57232\" (UID: \"d29997d2-a75c-4d10-a812-57d36aa57232\") " Jul 14 21:48:16.842089 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:16.842049 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29997d2-a75c-4d10-a812-57d36aa57232-kube-api-access-r7mmw" (OuterVolumeSpecName: "kube-api-access-r7mmw") pod "d29997d2-a75c-4d10-a812-57d36aa57232" (UID: "d29997d2-a75c-4d10-a812-57d36aa57232"). InnerVolumeSpecName "kube-api-access-r7mmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 21:48:16.851065 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:16.851023 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d29997d2-a75c-4d10-a812-57d36aa57232-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d29997d2-a75c-4d10-a812-57d36aa57232" (UID: "d29997d2-a75c-4d10-a812-57d36aa57232"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 21:48:16.941221 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:16.941118 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7mmw\" (UniqueName: \"kubernetes.io/projected/d29997d2-a75c-4d10-a812-57d36aa57232-kube-api-access-r7mmw\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:48:16.941221 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:16.941154 2552 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d29997d2-a75c-4d10-a812-57d36aa57232-tls-cert\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 21:48:17.362953 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:17.362915 2552 generic.go:356] "Generic (PLEG): container finished" podID="d29997d2-a75c-4d10-a812-57d36aa57232" containerID="65883e917af3f478f7648498590bc127626adb7352b025917c9fb2be238f2330" exitCode=0 Jul 14 21:48:17.362953 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:17.362950 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c7d9f96f9-kts62" Jul 14 21:48:17.363469 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:17.362986 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c7d9f96f9-kts62" event={"ID":"d29997d2-a75c-4d10-a812-57d36aa57232","Type":"ContainerDied","Data":"65883e917af3f478f7648498590bc127626adb7352b025917c9fb2be238f2330"} Jul 14 21:48:17.363469 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:17.363021 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c7d9f96f9-kts62" event={"ID":"d29997d2-a75c-4d10-a812-57d36aa57232","Type":"ContainerDied","Data":"b191c2ac57f548327618d2647e4c88d919cefac5850aa64d7feff95ad903b94a"} Jul 14 21:48:17.363469 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:17.363041 2552 scope.go:117] "RemoveContainer" containerID="65883e917af3f478f7648498590bc127626adb7352b025917c9fb2be238f2330" Jul 14 21:48:17.373268 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:17.373243 2552 scope.go:117] "RemoveContainer" containerID="65883e917af3f478f7648498590bc127626adb7352b025917c9fb2be238f2330" Jul 14 21:48:17.373612 ip-10-0-130-223 kubenswrapper[2552]: E0714 21:48:17.373587 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65883e917af3f478f7648498590bc127626adb7352b025917c9fb2be238f2330\": container with ID starting with 65883e917af3f478f7648498590bc127626adb7352b025917c9fb2be238f2330 not found: ID does not exist" containerID="65883e917af3f478f7648498590bc127626adb7352b025917c9fb2be238f2330" Jul 14 21:48:17.373778 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:17.373745 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65883e917af3f478f7648498590bc127626adb7352b025917c9fb2be238f2330"} err="failed to get container status \"65883e917af3f478f7648498590bc127626adb7352b025917c9fb2be238f2330\": rpc error: code = NotFound desc = could not find container \"65883e917af3f478f7648498590bc127626adb7352b025917c9fb2be238f2330\": container with ID starting with 65883e917af3f478f7648498590bc127626adb7352b025917c9fb2be238f2330 not found: ID does not exist" Jul 14 21:48:17.388101 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:17.388054 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c7d9f96f9-kts62"] Jul 14 21:48:17.392909 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:17.392867 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c7d9f96f9-kts62"] Jul 14 21:48:17.993039 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:17.993003 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d29997d2-a75c-4d10-a812-57d36aa57232" path="/var/lib/kubelet/pods/d29997d2-a75c-4d10-a812-57d36aa57232/volumes" Jul 14 21:48:39.261853 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:48:39.261806 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:49:28.170022 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:49:28.169987 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:49:30.180153 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:49:30.180109 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:50:11.473628 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:50:11.473586 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:50:11.564643 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:50:11.564605 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:50:11.701079 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:50:11.701041 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:50:18.456242 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:50:18.456180 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:50:18.564482 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:50:18.564441 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:50:18.666511 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:50:18.666475 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:50:18.851754 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:50:18.851711 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:50:25.226135 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:50:25.226097 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:50:25.293062 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:50:25.293011 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:51:17.573943 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:51:17.573903 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:51:31.390637 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:51:31.390595 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:52:13.972766 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:52:13.972735 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 21:52:13.974850 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:52:13.974819 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 21:52:13.979460 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:52:13.979438 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 21:52:13.981670 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:52:13.981650 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 21:53:04.386259 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:53:04.386227 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:53:04.446058 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:53:04.446020 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:53:25.334537 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:53:25.334499 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:57:14.010823 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:57:14.010792 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 21:57:14.014090 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:57:14.014065 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 21:57:14.017204 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:57:14.017170 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 21:57:14.020779 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:57:14.020762 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 21:57:14.456629 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:57:14.456594 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:57:14.539108 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:57:14.539072 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:57:21.873901 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:57:21.873856 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:59:55.353688 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:59:55.353595 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 21:59:56.776264 ip-10-0-130-223 kubenswrapper[2552]: I0714 21:59:56.776229 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:05.421236 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:05.421180 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:07.720955 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:07.720869 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:15.289332 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:15.289272 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:19.421769 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:19.421734 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:23.579375 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:23.579336 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:28.819946 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:28.819910 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:30.382914 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:30.382875 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:32.005094 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:32.005039 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:32.102515 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:32.102481 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:32.523927 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:32.523880 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:38.783406 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:38.783366 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:38.942138 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:38.942101 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:39.012708 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:39.012667 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:39.073842 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:39.073757 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:40.202644 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:40.202606 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:40.288305 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:40.288270 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:40.359541 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:40.359480 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:40.593177 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:40.593142 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:40.893493 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:40.893407 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:40.994117 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:40.994079 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:42.652006 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:42.651969 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:45.367205 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:45.367164 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:45.553813 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:45.553771 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:45.658956 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:45.658852 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:45.833884 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:45.833846 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:47.266516 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:47.266481 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:52.088432 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:52.088398 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:52.177820 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:52.177788 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:52.315824 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:52.315788 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:52.421257 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:52.421165 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:53.412342 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:53.412301 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:53.517002 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:53.516969 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:53.634457 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:53.634418 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:53.968314 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:53.968279 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:54.273829 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:54.273734 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:54.374821 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:54.374767 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:56.330530 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:56.330495 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:58.643262 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:58.643225 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:01:59.069812 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:01:59.069721 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:04.829741 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:04.829653 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:04.944059 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:04.944027 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:07.142837 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:07.142798 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:11.037044 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:11.037004 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:11.140915 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:11.140879 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:11.271234 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:11.271175 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:11.369807 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:11.369774 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:12.457518 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:12.457478 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:12.535444 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:12.535408 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:12.612530 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:12.612496 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:12.718931 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:12.718824 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:13.149391 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:13.149360 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:13.248939 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:13.248899 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:14.044389 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:14.044359 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 22:02:14.049571 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:14.049546 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 22:02:14.051227 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:14.051209 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 22:02:14.055902 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:14.055885 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 22:02:14.633213 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:14.633160 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:17.459064 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:17.459032 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:17.521211 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:17.521157 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:20.638430 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:20.638393 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:23.396078 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:23.396036 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:23.496741 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:23.496707 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:23.912341 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:23.912307 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:24.012674 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:24.012632 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:24.114038 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:24.113997 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:25.602031 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:25.601987 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:32.039915 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:32.039878 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:32.140134 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:32.140094 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:32.243980 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:32.243948 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:32.672707 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:32.672672 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:02:58.722727 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:02:58.722690 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:05.239131 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:05.239078 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:05.302860 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:05.302816 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:05.385600 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:05.385560 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:05.487375 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:05.487337 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:21.455800 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:21.455762 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:27.698051 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:27.698006 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:27.761706 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:27.761674 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:27.862184 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:27.862129 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:27.959017 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:27.958928 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:38.480074 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:38.480040 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:39.360848 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:39.360806 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:45.603828 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:45.603792 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:45.704130 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:45.704081 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:45.803280 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:45.803242 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:45.904718 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:45.904609 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:46.003378 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:46.003341 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:46.753603 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:46.753564 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:46.853767 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:46.853735 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:46.952979 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:46.952944 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:48.719930 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:48.719887 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:54.930483 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:54.930446 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:55.030761 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:55.030724 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:03:55.644914 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:03:55.644883 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:01.985105 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:01.985062 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:02.046158 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:02.046121 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:02.238128 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:02.238043 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:02.570001 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:02.569894 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:03.064654 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:03.064604 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:06.110248 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:06.110213 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:06.187455 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:06.187358 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:06.330492 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:06.330458 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:06.783719 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:06.783685 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:06.885681 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:06.885651 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:10.529256 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:10.529219 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:11.268705 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:11.268666 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:11.367760 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:11.367723 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:11.741605 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:11.741571 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:17.781093 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:17.781048 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:17.881380 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:17.881339 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:18.228976 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:18.228939 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:18.632550 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:18.632518 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:18.730871 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:18.730836 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:25.206745 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:25.206708 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:25.266651 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:25.266615 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:25.505335 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:25.505239 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:25.781886 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:25.781791 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:25.888693 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:25.888654 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:25.983711 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:25.983676 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:26.114169 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:26.114135 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:32.934016 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:32.933959 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:33.033677 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:33.033635 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:33.132619 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:33.132582 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:33.346034 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:33.346002 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:39.719841 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:39.719807 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:39.790171 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:39.790138 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:46.788209 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:46.788144 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:47.176780 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:47.176732 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:47.637159 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:47.637107 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:50.370398 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:50.370355 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:50.471383 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:50.471340 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:50.584715 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:50.584685 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:50.760039 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:50.759948 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:51.148676 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:51.148636 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:51.249625 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:51.249584 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:51.902432 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:51.902377 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:52.005122 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:52.005084 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:52.717778 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:52.717741 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:52.817982 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:52.817947 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:04:54.839085 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:04:54.839051 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:05:13.291449 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.291410 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc"] Jul 14 22:05:13.291868 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.291768 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29997d2-a75c-4d10-a812-57d36aa57232" containerName="authorino" Jul 14 22:05:13.291868 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.291779 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29997d2-a75c-4d10-a812-57d36aa57232" containerName="authorino" Jul 14 22:05:13.291868 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.291855 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d29997d2-a75c-4d10-a812-57d36aa57232" containerName="authorino" Jul 14 22:05:13.294946 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.294927 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.298756 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.298725 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f2113704\"/\"openshift-service-ca.crt\"" Jul 14 22:05:13.298905 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.298772 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f2113704\"/\"kube-root-ca.crt\"" Jul 14 22:05:13.298905 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.298772 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f2113704\"/\"default-dockercfg-hvmd4\"" Jul 14 22:05:13.298905 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.298772 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f2113704\"/\"e2e-lifecycle-model-f2113704-kserve-self-signed-certs\"" Jul 14 22:05:13.304225 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.304185 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc"] Jul 14 22:05:13.327659 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.327626 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:05:13.349317 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.349287 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-tls-certs\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.349499 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.349341 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-home\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.349499 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.349358 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-model-cache\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.349499 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.349429 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-tmp-dir\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.349622 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.349505 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nngj\" (UniqueName: \"kubernetes.io/projected/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-kube-api-access-5nngj\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.349622 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.349524 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-dshm\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.450530 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.450480 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-home\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.450530 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.450522 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-model-cache\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.450774 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.450554 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-tmp-dir\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.450774 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.450599 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nngj\" (UniqueName: \"kubernetes.io/projected/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-kube-api-access-5nngj\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.450774 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.450615 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-dshm\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.450774 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.450644 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-tls-certs\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.451007 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.450981 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-home\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.451072 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.450995 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-model-cache\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.451129 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.451076 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-tmp-dir\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.452911 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.452890 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-dshm\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.453201 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.453171 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-tls-certs\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.462342 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.462316 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nngj\" (UniqueName: \"kubernetes.io/projected/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-kube-api-access-5nngj\") pod \"e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.605903 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.605860 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:13.751301 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:13.751270 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc"] Jul 14 22:05:13.752638 ip-10-0-130-223 kubenswrapper[2552]: W0714 22:05:13.752618 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1534ca1_16d7_4c12_88b4_7de7a7c41cc9.slice/crio-57ce434cbb8323d1f08285d1c7056ad583ddfecd41efb25bc91a6a5cb3c08286 WatchSource:0}: Error finding container 57ce434cbb8323d1f08285d1c7056ad583ddfecd41efb25bc91a6a5cb3c08286: Status 404 returned error can't find the container with id 57ce434cbb8323d1f08285d1c7056ad583ddfecd41efb25bc91a6a5cb3c08286 Jul 14 22:05:14.742104 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:14.742072 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" event={"ID":"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9","Type":"ContainerStarted","Data":"db07e726c6eb001207e0074e71ae678aab9ed81c1aefac27efca5882af56668d"} Jul 14 22:05:14.742498 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:14.742112 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" event={"ID":"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9","Type":"ContainerStarted","Data":"57ce434cbb8323d1f08285d1c7056ad583ddfecd41efb25bc91a6a5cb3c08286"} Jul 14 22:05:14.763677 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:14.763626 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" podStartSLOduration=1.763609513 podStartE2EDuration="1.763609513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:05:14.762828644 +0000 UTC m=+1681.301534317" watchObservedRunningTime="2026-07-14 22:05:14.763609513 +0000 UTC m=+1681.302315184" Jul 14 22:05:23.606966 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:23.606901 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:23.606966 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:23.606977 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:23.619408 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:23.619384 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:23.799247 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:23.799217 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:33.218886 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:33.218845 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:05:33.715426 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:33.715389 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:05:39.376427 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:39.376339 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:05:44.363032 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:44.362988 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc"] Jul 14 22:05:44.422640 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:44.422597 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:05:59.383399 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.383338 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" podUID="d1534ca1-16d7-4c12-88b4-7de7a7c41cc9" containerName="main" containerID="cri-o://db07e726c6eb001207e0074e71ae678aab9ed81c1aefac27efca5882af56668d" gracePeriod=45 Jul 14 22:05:59.631905 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.631874 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:59.656702 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.656601 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nngj\" (UniqueName: \"kubernetes.io/projected/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-kube-api-access-5nngj\") pod \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " Jul 14 22:05:59.656702 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.656688 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-tmp-dir\") pod \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " Jul 14 22:05:59.656923 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.656737 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-model-cache\") pod \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " Jul 14 22:05:59.657081 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.657055 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d1534ca1-16d7-4c12-88b4-7de7a7c41cc9" (UID: "d1534ca1-16d7-4c12-88b4-7de7a7c41cc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:05:59.657155 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.657095 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1534ca1-16d7-4c12-88b4-7de7a7c41cc9" (UID: "d1534ca1-16d7-4c12-88b4-7de7a7c41cc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:05:59.659529 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.659484 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-kube-api-access-5nngj" (OuterVolumeSpecName: "kube-api-access-5nngj") pod "d1534ca1-16d7-4c12-88b4-7de7a7c41cc9" (UID: "d1534ca1-16d7-4c12-88b4-7de7a7c41cc9"). InnerVolumeSpecName "kube-api-access-5nngj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:05:59.757304 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.757260 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-tls-certs\") pod \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " Jul 14 22:05:59.757304 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.757305 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-dshm\") pod \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " Jul 14 22:05:59.757509 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.757324 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-home\") pod \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\" (UID: \"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9\") " Jul 14 22:05:59.757625 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.757606 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-model-cache\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:05:59.757665 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.757632 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nngj\" (UniqueName: \"kubernetes.io/projected/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-kube-api-access-5nngj\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:05:59.757665 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.757636 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-home" (OuterVolumeSpecName: "home") pod "d1534ca1-16d7-4c12-88b4-7de7a7c41cc9" (UID: "d1534ca1-16d7-4c12-88b4-7de7a7c41cc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:05:59.757665 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.757646 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-tmp-dir\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:05:59.759407 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.759374 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-dshm" (OuterVolumeSpecName: "dshm") pod "d1534ca1-16d7-4c12-88b4-7de7a7c41cc9" (UID: "d1534ca1-16d7-4c12-88b4-7de7a7c41cc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:05:59.759407 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.759378 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1534ca1-16d7-4c12-88b4-7de7a7c41cc9" (UID: "d1534ca1-16d7-4c12-88b4-7de7a7c41cc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:05:59.858869 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.858833 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-tls-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:05:59.858869 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.858861 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-dshm\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:05:59.858869 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.858869 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9-home\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:05:59.973915 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.973819 2552 generic.go:356] "Generic (PLEG): container finished" podID="d1534ca1-16d7-4c12-88b4-7de7a7c41cc9" containerID="db07e726c6eb001207e0074e71ae678aab9ed81c1aefac27efca5882af56668d" exitCode=0 Jul 14 22:05:59.973915 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.973863 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" Jul 14 22:05:59.973915 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.973901 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" event={"ID":"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9","Type":"ContainerDied","Data":"db07e726c6eb001207e0074e71ae678aab9ed81c1aefac27efca5882af56668d"} Jul 14 22:05:59.974169 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.973940 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc" event={"ID":"d1534ca1-16d7-4c12-88b4-7de7a7c41cc9","Type":"ContainerDied","Data":"57ce434cbb8323d1f08285d1c7056ad583ddfecd41efb25bc91a6a5cb3c08286"} Jul 14 22:05:59.974169 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.973961 2552 scope.go:117] "RemoveContainer" containerID="db07e726c6eb001207e0074e71ae678aab9ed81c1aefac27efca5882af56668d" Jul 14 22:05:59.983441 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.983425 2552 scope.go:117] "RemoveContainer" containerID="db07e726c6eb001207e0074e71ae678aab9ed81c1aefac27efca5882af56668d" Jul 14 22:05:59.983725 ip-10-0-130-223 kubenswrapper[2552]: E0714 22:05:59.983707 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db07e726c6eb001207e0074e71ae678aab9ed81c1aefac27efca5882af56668d\": container with ID starting with db07e726c6eb001207e0074e71ae678aab9ed81c1aefac27efca5882af56668d not found: ID does not exist" containerID="db07e726c6eb001207e0074e71ae678aab9ed81c1aefac27efca5882af56668d" Jul 14 22:05:59.983770 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.983735 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db07e726c6eb001207e0074e71ae678aab9ed81c1aefac27efca5882af56668d"} err="failed to get container status \"db07e726c6eb001207e0074e71ae678aab9ed81c1aefac27efca5882af56668d\": rpc error: code = NotFound desc = could not find container \"db07e726c6eb001207e0074e71ae678aab9ed81c1aefac27efca5882af56668d\": container with ID starting with db07e726c6eb001207e0074e71ae678aab9ed81c1aefac27efca5882af56668d not found: ID does not exist" Jul 14 22:05:59.999386 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:05:59.999357 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc"] Jul 14 22:06:00.004600 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:00.004576 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f2113704/e2e-lifecycle-model-f2113704-kserve-d8dfffd7-ntscc"] Jul 14 22:06:01.991365 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:01.991333 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1534ca1-16d7-4c12-88b4-7de7a7c41cc9" path="/var/lib/kubelet/pods/d1534ca1-16d7-4c12-88b4-7de7a7c41cc9/volumes" Jul 14 22:06:16.962953 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:16.962914 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:20.711658 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:20.711617 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:21.955381 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:21.955333 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:28.167236 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:28.167168 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:28.231968 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:28.231931 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:28.491566 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:28.491467 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:28.730829 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:28.730774 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:29.850528 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:29.850495 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:29.951744 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:29.951696 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:30.051053 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:30.051019 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:30.890786 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:30.890742 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:30.995140 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:30.995109 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:31.095199 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:31.095161 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:32.572256 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:32.572217 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:39.151392 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:39.151355 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:39.252401 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:39.252366 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:39.864220 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:39.864168 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:46.177172 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:46.177131 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:46.280321 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:46.280279 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:46.377019 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:46.376984 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:56.972072 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:56.972027 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:06:57.333845 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:06:57.333751 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:14.078641 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:14.078609 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 22:07:14.085577 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:14.085551 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 22:07:14.085749 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:14.085555 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 22:07:14.092425 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:14.092399 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 22:07:18.378153 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:18.378119 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:24.717304 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:24.717270 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:24.817765 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:24.817731 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:25.105734 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:25.105688 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:25.444878 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:25.444789 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:25.979857 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:25.979825 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:28.073813 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:28.073772 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:28.188411 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:28.188376 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:28.283375 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:28.283337 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:28.386235 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:28.386170 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:28.684224 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:28.684111 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:28.785084 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:28.785046 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:32.618000 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:32.617961 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:32.984711 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:32.984598 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:33.325504 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:33.325405 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:33.563042 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:33.563003 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:39.169725 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:39.169685 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:39.265427 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:39.265380 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:39.422482 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:39.422381 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:39.598491 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:39.598459 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:40.093448 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:40.093410 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:40.204248 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:40.204214 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:43.732318 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:43.732283 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:49.566775 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:49.566740 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:07:49.665876 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:07:49.665843 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:08:00.926783 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:00.926741 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:08:01.025158 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:01.025117 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:08:03.195552 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.195518 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z"] Jul 14 22:08:03.195933 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.195874 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1534ca1-16d7-4c12-88b4-7de7a7c41cc9" containerName="main" Jul 14 22:08:03.195933 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.195885 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1534ca1-16d7-4c12-88b4-7de7a7c41cc9" containerName="main" Jul 14 22:08:03.196005 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.195941 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1534ca1-16d7-4c12-88b4-7de7a7c41cc9" containerName="main" Jul 14 22:08:03.198982 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.198964 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.202890 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.202864 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f62700\"/\"openshift-service-ca.crt\"" Jul 14 22:08:03.202890 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.202889 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f62700\"/\"kube-root-ca.crt\"" Jul 14 22:08:03.203083 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.202863 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f62700\"/\"test-model-f62700-kserve-self-signed-certs\"" Jul 14 22:08:03.203083 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.202862 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f62700\"/\"default-dockercfg-k86kj\"" Jul 14 22:08:03.208765 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.208745 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z"] Jul 14 22:08:03.214888 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.214866 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:08:03.278076 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.278032 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-tmp-dir\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.278076 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.278081 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-home\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.278380 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.278139 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c760164-6622-49f6-b259-f07c3350b68f-tls-certs\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.278380 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.278172 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrzph\" (UniqueName: \"kubernetes.io/projected/5c760164-6622-49f6-b259-f07c3350b68f-kube-api-access-rrzph\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.278380 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.278278 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-dshm\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.278380 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.278325 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-model-cache\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.379081 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.379039 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c760164-6622-49f6-b259-f07c3350b68f-tls-certs\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.379281 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.379089 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrzph\" (UniqueName: \"kubernetes.io/projected/5c760164-6622-49f6-b259-f07c3350b68f-kube-api-access-rrzph\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.379281 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.379167 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-dshm\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.379412 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.379320 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-model-cache\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.379412 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.379397 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-tmp-dir\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.379520 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.379434 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-home\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.379704 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.379685 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-model-cache\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.379794 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.379769 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-tmp-dir\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.379860 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.379808 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-home\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.381516 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.381496 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-dshm\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.381642 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.381627 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c760164-6622-49f6-b259-f07c3350b68f-tls-certs\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.390964 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.390933 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrzph\" (UniqueName: \"kubernetes.io/projected/5c760164-6622-49f6-b259-f07c3350b68f-kube-api-access-rrzph\") pod \"test-model-f62700-kserve-85bc8898ff-74v7z\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.509480 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.509380 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:03.647068 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:03.647039 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z"] Jul 14 22:08:03.648580 ip-10-0-130-223 kubenswrapper[2552]: W0714 22:08:03.648557 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c760164_6622_49f6_b259_f07c3350b68f.slice/crio-2c503c7157ed116d7c79a3b60efb1a1c62bfbc5d92b25328388d26c1741d2800 WatchSource:0}: Error finding container 2c503c7157ed116d7c79a3b60efb1a1c62bfbc5d92b25328388d26c1741d2800: Status 404 returned error can't find the container with id 2c503c7157ed116d7c79a3b60efb1a1c62bfbc5d92b25328388d26c1741d2800 Jul 14 22:08:04.623845 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:04.623809 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" event={"ID":"5c760164-6622-49f6-b259-f07c3350b68f","Type":"ContainerStarted","Data":"420fc0af64c7ebc048622c0ba57d6cbe74343d9b801df59e2cb5df18df8f3379"} Jul 14 22:08:04.623845 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:04.623851 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" event={"ID":"5c760164-6622-49f6-b259-f07c3350b68f","Type":"ContainerStarted","Data":"2c503c7157ed116d7c79a3b60efb1a1c62bfbc5d92b25328388d26c1741d2800"} Jul 14 22:08:04.643898 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:04.643845 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" podStartSLOduration=1.643830047 podStartE2EDuration="1.643830047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:08:04.642446828 +0000 UTC m=+1851.181152514" watchObservedRunningTime="2026-07-14 22:08:04.643830047 +0000 UTC m=+1851.182535719" Jul 14 22:08:09.928972 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:09.928934 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:08:13.509964 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:13.509916 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:13.510416 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:13.509977 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:13.522721 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:13.522698 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:13.688758 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:13.688730 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:08:34.997979 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:34.997944 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:08:51.093053 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:51.092964 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:08:55.567402 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:08:55.567367 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:02.529727 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:02.529658 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:02.631544 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:02.631503 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:13.412754 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:13.412718 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z"] Jul 14 22:09:13.434110 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:13.434082 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:14.421488 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:14.421453 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:14.591782 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:14.591745 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:15.007928 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:15.007899 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:15.110116 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:15.110077 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:15.209063 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:15.209034 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:15.299797 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:15.299707 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:20.085928 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:20.085888 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:28.433068 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.433004 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" podUID="5c760164-6622-49f6-b259-f07c3350b68f" containerName="main" containerID="cri-o://420fc0af64c7ebc048622c0ba57d6cbe74343d9b801df59e2cb5df18df8f3379" gracePeriod=45 Jul 14 22:09:28.683467 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.683404 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:09:28.833797 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.833761 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-dshm\") pod \"5c760164-6622-49f6-b259-f07c3350b68f\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " Jul 14 22:09:28.833992 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.833840 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-home\") pod \"5c760164-6622-49f6-b259-f07c3350b68f\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " Jul 14 22:09:28.833992 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.833873 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c760164-6622-49f6-b259-f07c3350b68f-tls-certs\") pod \"5c760164-6622-49f6-b259-f07c3350b68f\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " Jul 14 22:09:28.833992 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.833902 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-tmp-dir\") pod \"5c760164-6622-49f6-b259-f07c3350b68f\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " Jul 14 22:09:28.833992 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.833919 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-model-cache\") pod \"5c760164-6622-49f6-b259-f07c3350b68f\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " Jul 14 22:09:28.833992 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.833959 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrzph\" (UniqueName: \"kubernetes.io/projected/5c760164-6622-49f6-b259-f07c3350b68f-kube-api-access-rrzph\") pod \"5c760164-6622-49f6-b259-f07c3350b68f\" (UID: \"5c760164-6622-49f6-b259-f07c3350b68f\") " Jul 14 22:09:28.834309 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.834185 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c760164-6622-49f6-b259-f07c3350b68f" (UID: "5c760164-6622-49f6-b259-f07c3350b68f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:09:28.834309 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.834246 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-home" (OuterVolumeSpecName: "home") pod "5c760164-6622-49f6-b259-f07c3350b68f" (UID: "5c760164-6622-49f6-b259-f07c3350b68f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:09:28.834422 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.834301 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c760164-6622-49f6-b259-f07c3350b68f" (UID: "5c760164-6622-49f6-b259-f07c3350b68f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:09:28.836112 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.836081 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-dshm" (OuterVolumeSpecName: "dshm") pod "5c760164-6622-49f6-b259-f07c3350b68f" (UID: "5c760164-6622-49f6-b259-f07c3350b68f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:09:28.836257 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.836161 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c760164-6622-49f6-b259-f07c3350b68f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c760164-6622-49f6-b259-f07c3350b68f" (UID: "5c760164-6622-49f6-b259-f07c3350b68f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:09:28.836257 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.836212 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c760164-6622-49f6-b259-f07c3350b68f-kube-api-access-rrzph" (OuterVolumeSpecName: "kube-api-access-rrzph") pod "5c760164-6622-49f6-b259-f07c3350b68f" (UID: "5c760164-6622-49f6-b259-f07c3350b68f"). InnerVolumeSpecName "kube-api-access-rrzph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:09:28.935109 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.935012 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c760164-6622-49f6-b259-f07c3350b68f-tls-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:09:28.935109 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.935043 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-tmp-dir\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:09:28.935109 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.935055 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-model-cache\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:09:28.935109 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.935066 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrzph\" (UniqueName: \"kubernetes.io/projected/5c760164-6622-49f6-b259-f07c3350b68f-kube-api-access-rrzph\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:09:28.935109 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.935077 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-dshm\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:09:28.935109 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:28.935087 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c760164-6622-49f6-b259-f07c3350b68f-home\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:09:29.071540 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:29.071507 2552 generic.go:356] "Generic (PLEG): container finished" podID="5c760164-6622-49f6-b259-f07c3350b68f" containerID="420fc0af64c7ebc048622c0ba57d6cbe74343d9b801df59e2cb5df18df8f3379" exitCode=0 Jul 14 22:09:29.071748 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:29.071553 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" Jul 14 22:09:29.071748 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:29.071596 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" event={"ID":"5c760164-6622-49f6-b259-f07c3350b68f","Type":"ContainerDied","Data":"420fc0af64c7ebc048622c0ba57d6cbe74343d9b801df59e2cb5df18df8f3379"} Jul 14 22:09:29.071748 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:29.071654 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z" event={"ID":"5c760164-6622-49f6-b259-f07c3350b68f","Type":"ContainerDied","Data":"2c503c7157ed116d7c79a3b60efb1a1c62bfbc5d92b25328388d26c1741d2800"} Jul 14 22:09:29.071748 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:29.071676 2552 scope.go:117] "RemoveContainer" containerID="420fc0af64c7ebc048622c0ba57d6cbe74343d9b801df59e2cb5df18df8f3379" Jul 14 22:09:29.081586 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:29.081563 2552 scope.go:117] "RemoveContainer" containerID="420fc0af64c7ebc048622c0ba57d6cbe74343d9b801df59e2cb5df18df8f3379" Jul 14 22:09:29.081875 ip-10-0-130-223 kubenswrapper[2552]: E0714 22:09:29.081845 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"420fc0af64c7ebc048622c0ba57d6cbe74343d9b801df59e2cb5df18df8f3379\": container with ID starting with 420fc0af64c7ebc048622c0ba57d6cbe74343d9b801df59e2cb5df18df8f3379 not found: ID does not exist" containerID="420fc0af64c7ebc048622c0ba57d6cbe74343d9b801df59e2cb5df18df8f3379" Jul 14 22:09:29.081960 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:29.081881 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"420fc0af64c7ebc048622c0ba57d6cbe74343d9b801df59e2cb5df18df8f3379"} err="failed to get container status \"420fc0af64c7ebc048622c0ba57d6cbe74343d9b801df59e2cb5df18df8f3379\": rpc error: code = NotFound desc = could not find container \"420fc0af64c7ebc048622c0ba57d6cbe74343d9b801df59e2cb5df18df8f3379\": container with ID starting with 420fc0af64c7ebc048622c0ba57d6cbe74343d9b801df59e2cb5df18df8f3379 not found: ID does not exist" Jul 14 22:09:29.093447 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:29.093419 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z"] Jul 14 22:09:29.097248 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:29.097225 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f62700/test-model-f62700-kserve-85bc8898ff-74v7z"] Jul 14 22:09:29.992509 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:29.992473 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c760164-6622-49f6-b259-f07c3350b68f" path="/var/lib/kubelet/pods/5c760164-6622-49f6-b259-f07c3350b68f/volumes" Jul 14 22:09:51.716376 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:51.716339 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:52.675657 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:52.675604 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:52.773864 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:52.773832 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:58.145126 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:58.145085 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:58.458556 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:58.458474 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:58.702437 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:58.702398 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:58.953826 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:58.953797 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:59.373821 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:59.373770 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:59.473972 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:59.473931 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:59.573776 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:59.573736 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:59.678498 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:09:59.678406 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:05.825137 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:05.825101 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:05.925917 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:05.925876 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:06.120855 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:06.120816 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:06.361115 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:06.361028 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:06.728979 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:06.728945 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:07.069675 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:07.069586 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:07.151760 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:07.151728 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:09.924101 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:09.924063 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:10.021409 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:10.021374 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:10.125081 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:10.125044 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:10.301940 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:10.301847 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:10.606388 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:10.606355 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:10.706943 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:10.706904 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:14.494502 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:14.494408 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:14.996324 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:14.996286 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:16.239241 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:16.239206 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:17.850460 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:17.850425 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:17.927602 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:17.927542 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:18.201056 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:18.200964 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:18.302035 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:18.301996 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:18.459968 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:18.459869 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:18.559097 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:18.559061 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:22.384258 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:22.384220 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:30.767894 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:30.767849 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:31.193489 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:31.193453 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:31.294201 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:31.294150 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:31.393567 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:31.393520 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:31.548747 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:31.548661 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:37.607295 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:37.607253 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:37.712804 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:37.712770 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:37.904285 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:37.904176 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:38.159162 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:38.159070 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:38.472930 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:38.472840 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:38.871288 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:38.871249 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:38.970935 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:38.970898 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:39.070732 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:39.070700 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:39.248067 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:39.247981 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:45.318553 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:45.318520 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:45.384729 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:45.384683 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:45.605521 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:45.605482 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:45.842769 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:45.842733 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:46.198347 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:46.198307 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:46.538402 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:46.538252 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:46.692046 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:46.692010 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:49.207429 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:49.207391 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:49.305155 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:49.305117 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:49.375180 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:49.375140 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:49.838940 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:49.838903 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:49.939103 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:49.939068 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:50.039587 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:50.039553 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:54.013798 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:54.013758 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:54.114165 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:54.114130 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:54.481680 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:54.481647 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:54.712084 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:54.712047 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:57.129064 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:57.129030 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:57.222772 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:57.222738 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:57.302887 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:57.302848 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:57.818906 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:57.818868 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:57.918777 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:57.918745 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:58.020112 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:10:58.020072 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:11:01.607361 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:01.607324 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:11:10.810632 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:10.810596 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:11:10.910456 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:10.910414 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:11:18.729894 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:18.729858 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:11:18.833543 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:18.833500 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:11:22.079357 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.079322 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97"] Jul 14 22:11:22.079748 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.079656 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c760164-6622-49f6-b259-f07c3350b68f" containerName="main" Jul 14 22:11:22.079748 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.079666 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c760164-6622-49f6-b259-f07c3350b68f" containerName="main" Jul 14 22:11:22.079748 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.079732 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c760164-6622-49f6-b259-f07c3350b68f" containerName="main" Jul 14 22:11:22.082684 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.082665 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.085537 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.085514 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e8c0aa\"/\"openshift-service-ca.crt\"" Jul 14 22:11:22.085672 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.085587 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e8c0aa\"/\"kube-root-ca.crt\"" Jul 14 22:11:22.086819 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.086803 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e8c0aa\"/\"default-dockercfg-wr5pv\"" Jul 14 22:11:22.086894 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.086877 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e8c0aa\"/\"auth-test-model-e8c0aa-kserve-self-signed-certs\"" Jul 14 22:11:22.097780 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.097749 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97"] Jul 14 22:11:22.110771 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.110736 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:11:22.214398 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.214359 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1569c24-f138-49e1-83e2-f90d2a6e433c-tls-certs\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.214398 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.214404 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-home\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.214623 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.214431 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-dshm\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.214623 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.214556 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-model-cache\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.214623 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.214596 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4tqc\" (UniqueName: \"kubernetes.io/projected/f1569c24-f138-49e1-83e2-f90d2a6e433c-kube-api-access-d4tqc\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.214623 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.214620 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-tmp-dir\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.315829 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.315770 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-home\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.316020 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.315855 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-dshm\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.316020 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.315984 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-model-cache\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.316102 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.316037 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4tqc\" (UniqueName: \"kubernetes.io/projected/f1569c24-f138-49e1-83e2-f90d2a6e433c-kube-api-access-d4tqc\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.316102 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.316076 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-tmp-dir\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.316181 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.316159 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1569c24-f138-49e1-83e2-f90d2a6e433c-tls-certs\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.316283 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.316227 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-home\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.316430 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.316406 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-model-cache\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.316525 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.316505 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-tmp-dir\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.318227 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.318203 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-dshm\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.318532 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.318517 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1569c24-f138-49e1-83e2-f90d2a6e433c-tls-certs\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.325423 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.325402 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4tqc\" (UniqueName: \"kubernetes.io/projected/f1569c24-f138-49e1-83e2-f90d2a6e433c-kube-api-access-d4tqc\") pod \"auth-test-model-e8c0aa-kserve-6b8578b79-dxx97\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.391543 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.391511 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:22.526701 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.526673 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97"] Jul 14 22:11:22.529017 ip-10-0-130-223 kubenswrapper[2552]: W0714 22:11:22.528989 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1569c24_f138_49e1_83e2_f90d2a6e433c.slice/crio-f2920786b2203d15c9596f4899ac9cf19a763953ef65a674d2d876a34f342018 WatchSource:0}: Error finding container f2920786b2203d15c9596f4899ac9cf19a763953ef65a674d2d876a34f342018: Status 404 returned error can't find the container with id f2920786b2203d15c9596f4899ac9cf19a763953ef65a674d2d876a34f342018 Jul 14 22:11:22.663213 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.663093 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" event={"ID":"f1569c24-f138-49e1-83e2-f90d2a6e433c","Type":"ContainerStarted","Data":"b3bd5f1255d846949b0959ef47c2358826e9485b70d811718cebc53cfaffc888"} Jul 14 22:11:22.663213 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.663140 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" event={"ID":"f1569c24-f138-49e1-83e2-f90d2a6e433c","Type":"ContainerStarted","Data":"f2920786b2203d15c9596f4899ac9cf19a763953ef65a674d2d876a34f342018"} Jul 14 22:11:22.682789 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:22.682736 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" podStartSLOduration=0.682721206 podStartE2EDuration="682.721206ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:11:22.68072497 +0000 UTC m=+2049.219430643" watchObservedRunningTime="2026-07-14 22:11:22.682721206 +0000 UTC m=+2049.221426877" Jul 14 22:11:32.392627 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:32.392568 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:32.392627 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:32.392630 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:32.405335 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:32.405303 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:32.726605 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:32.726518 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:11:51.122299 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:11:51.122180 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:12:14.116347 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:12:14.116313 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 22:12:14.123667 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:12:14.123638 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 22:12:14.123898 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:12:14.123873 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 22:12:14.130778 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:12:14.130758 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 22:13:03.128649 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:03.128611 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:13:48.924212 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:48.924155 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:13:52.006709 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:52.006671 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97"] Jul 14 22:13:52.027014 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:52.026979 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:13:52.830751 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:52.830716 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:13:59.606308 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.606273 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq"] Jul 14 22:13:59.609727 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.609709 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.612502 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.612476 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e8c0aa\"/\"sub-test-model-e8c0aa-kserve-self-signed-certs\"" Jul 14 22:13:59.627587 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.627560 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq"] Jul 14 22:13:59.674657 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.674622 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-home\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.674841 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.674666 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-tmp-dir\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.674841 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.674747 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c782097-ec32-459f-a955-2572be3b056e-tls-certs\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.674841 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.674784 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-model-cache\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.674841 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.674804 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrkfq\" (UniqueName: \"kubernetes.io/projected/4c782097-ec32-459f-a955-2572be3b056e-kube-api-access-zrkfq\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.675025 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.674852 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-dshm\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.709106 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.709067 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:13:59.775348 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.775308 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-tmp-dir\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.775544 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.775392 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c782097-ec32-459f-a955-2572be3b056e-tls-certs\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.775544 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.775450 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-model-cache\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.775544 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.775484 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrkfq\" (UniqueName: \"kubernetes.io/projected/4c782097-ec32-459f-a955-2572be3b056e-kube-api-access-zrkfq\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.775544 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.775508 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-dshm\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.775544 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.775538 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-home\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.775817 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.775779 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-tmp-dir\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.775883 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.775811 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-model-cache\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.775943 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.775930 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-home\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.777834 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.777804 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-dshm\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.778008 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.777991 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c782097-ec32-459f-a955-2572be3b056e-tls-certs\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.787626 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.787603 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrkfq\" (UniqueName: \"kubernetes.io/projected/4c782097-ec32-459f-a955-2572be3b056e-kube-api-access-zrkfq\") pod \"sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:13:59.919290 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:13:59.919168 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:14:00.065662 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:00.065634 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq"] Jul 14 22:14:00.067566 ip-10-0-130-223 kubenswrapper[2552]: W0714 22:14:00.067535 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c782097_ec32_459f_a955_2572be3b056e.slice/crio-6b1ef72bd2593321489bd9b2214d223af80276088d7a6d433c1cafba5bde321a WatchSource:0}: Error finding container 6b1ef72bd2593321489bd9b2214d223af80276088d7a6d433c1cafba5bde321a: Status 404 returned error can't find the container with id 6b1ef72bd2593321489bd9b2214d223af80276088d7a6d433c1cafba5bde321a Jul 14 22:14:00.481917 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:00.481879 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" event={"ID":"4c782097-ec32-459f-a955-2572be3b056e","Type":"ContainerStarted","Data":"60166de3e9f5fb6fb2c2f2a8846e9919b153fb25df3e2acbbff394349db066cd"} Jul 14 22:14:00.481917 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:00.481923 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" event={"ID":"4c782097-ec32-459f-a955-2572be3b056e","Type":"ContainerStarted","Data":"6b1ef72bd2593321489bd9b2214d223af80276088d7a6d433c1cafba5bde321a"} Jul 14 22:14:00.508083 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:00.508024 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" podStartSLOduration=1.5080084440000001 podStartE2EDuration="1.508008444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:14:00.506980941 +0000 UTC m=+2207.045686614" watchObservedRunningTime="2026-07-14 22:14:00.508008444 +0000 UTC m=+2207.046714115" Jul 14 22:14:07.026975 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.026923 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" podUID="f1569c24-f138-49e1-83e2-f90d2a6e433c" containerName="main" containerID="cri-o://b3bd5f1255d846949b0959ef47c2358826e9485b70d811718cebc53cfaffc888" gracePeriod=45 Jul 14 22:14:07.282552 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.282487 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:14:07.343707 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.343673 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-model-cache\") pod \"f1569c24-f138-49e1-83e2-f90d2a6e433c\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " Jul 14 22:14:07.343899 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.343725 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1569c24-f138-49e1-83e2-f90d2a6e433c-tls-certs\") pod \"f1569c24-f138-49e1-83e2-f90d2a6e433c\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " Jul 14 22:14:07.343899 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.343762 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4tqc\" (UniqueName: \"kubernetes.io/projected/f1569c24-f138-49e1-83e2-f90d2a6e433c-kube-api-access-d4tqc\") pod \"f1569c24-f138-49e1-83e2-f90d2a6e433c\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " Jul 14 22:14:07.343899 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.343782 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-dshm\") pod \"f1569c24-f138-49e1-83e2-f90d2a6e433c\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " Jul 14 22:14:07.343899 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.343814 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-home\") pod \"f1569c24-f138-49e1-83e2-f90d2a6e433c\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " Jul 14 22:14:07.343899 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.343840 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-tmp-dir\") pod \"f1569c24-f138-49e1-83e2-f90d2a6e433c\" (UID: \"f1569c24-f138-49e1-83e2-f90d2a6e433c\") " Jul 14 22:14:07.344177 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.344005 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1569c24-f138-49e1-83e2-f90d2a6e433c" (UID: "f1569c24-f138-49e1-83e2-f90d2a6e433c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:14:07.344177 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.344132 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-home" (OuterVolumeSpecName: "home") pod "f1569c24-f138-49e1-83e2-f90d2a6e433c" (UID: "f1569c24-f138-49e1-83e2-f90d2a6e433c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:14:07.344177 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.344155 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-model-cache\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:14:07.344177 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.344156 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1569c24-f138-49e1-83e2-f90d2a6e433c" (UID: "f1569c24-f138-49e1-83e2-f90d2a6e433c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:14:07.346091 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.346062 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-dshm" (OuterVolumeSpecName: "dshm") pod "f1569c24-f138-49e1-83e2-f90d2a6e433c" (UID: "f1569c24-f138-49e1-83e2-f90d2a6e433c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:14:07.346183 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.346155 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1569c24-f138-49e1-83e2-f90d2a6e433c-kube-api-access-d4tqc" (OuterVolumeSpecName: "kube-api-access-d4tqc") pod "f1569c24-f138-49e1-83e2-f90d2a6e433c" (UID: "f1569c24-f138-49e1-83e2-f90d2a6e433c"). InnerVolumeSpecName "kube-api-access-d4tqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:14:07.346183 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.346161 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1569c24-f138-49e1-83e2-f90d2a6e433c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1569c24-f138-49e1-83e2-f90d2a6e433c" (UID: "f1569c24-f138-49e1-83e2-f90d2a6e433c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:14:07.444939 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.444895 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-home\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:14:07.444939 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.444931 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-tmp-dir\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:14:07.444939 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.444942 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1569c24-f138-49e1-83e2-f90d2a6e433c-tls-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:14:07.445181 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.444950 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4tqc\" (UniqueName: \"kubernetes.io/projected/f1569c24-f138-49e1-83e2-f90d2a6e433c-kube-api-access-d4tqc\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:14:07.445181 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.444961 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1569c24-f138-49e1-83e2-f90d2a6e433c-dshm\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:14:07.518898 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.518854 2552 generic.go:356] "Generic (PLEG): container finished" podID="f1569c24-f138-49e1-83e2-f90d2a6e433c" containerID="b3bd5f1255d846949b0959ef47c2358826e9485b70d811718cebc53cfaffc888" exitCode=0 Jul 14 22:14:07.518898 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.518901 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" Jul 14 22:14:07.519114 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.518942 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" event={"ID":"f1569c24-f138-49e1-83e2-f90d2a6e433c","Type":"ContainerDied","Data":"b3bd5f1255d846949b0959ef47c2358826e9485b70d811718cebc53cfaffc888"} Jul 14 22:14:07.519114 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.518995 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97" event={"ID":"f1569c24-f138-49e1-83e2-f90d2a6e433c","Type":"ContainerDied","Data":"f2920786b2203d15c9596f4899ac9cf19a763953ef65a674d2d876a34f342018"} Jul 14 22:14:07.519114 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.519017 2552 scope.go:117] "RemoveContainer" containerID="b3bd5f1255d846949b0959ef47c2358826e9485b70d811718cebc53cfaffc888" Jul 14 22:14:07.528678 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.528660 2552 scope.go:117] "RemoveContainer" containerID="b3bd5f1255d846949b0959ef47c2358826e9485b70d811718cebc53cfaffc888" Jul 14 22:14:07.528959 ip-10-0-130-223 kubenswrapper[2552]: E0714 22:14:07.528938 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3bd5f1255d846949b0959ef47c2358826e9485b70d811718cebc53cfaffc888\": container with ID starting with b3bd5f1255d846949b0959ef47c2358826e9485b70d811718cebc53cfaffc888 not found: ID does not exist" containerID="b3bd5f1255d846949b0959ef47c2358826e9485b70d811718cebc53cfaffc888" Jul 14 22:14:07.529040 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.528965 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3bd5f1255d846949b0959ef47c2358826e9485b70d811718cebc53cfaffc888"} err="failed to get container status \"b3bd5f1255d846949b0959ef47c2358826e9485b70d811718cebc53cfaffc888\": rpc error: code = NotFound desc = could not find container \"b3bd5f1255d846949b0959ef47c2358826e9485b70d811718cebc53cfaffc888\": container with ID starting with b3bd5f1255d846949b0959ef47c2358826e9485b70d811718cebc53cfaffc888 not found: ID does not exist" Jul 14 22:14:07.542140 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.542079 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97"] Jul 14 22:14:07.548891 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.548863 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/auth-test-model-e8c0aa-kserve-6b8578b79-dxx97"] Jul 14 22:14:07.992567 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:07.992535 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1569c24-f138-49e1-83e2-f90d2a6e433c" path="/var/lib/kubelet/pods/f1569c24-f138-49e1-83e2-f90d2a6e433c/volumes" Jul 14 22:14:09.919603 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:09.919563 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:14:09.920084 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:09.919618 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:14:09.932666 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:09.932631 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:14:10.547220 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:10.547175 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:14:26.291495 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.291462 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj"] Jul 14 22:14:26.291881 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.291822 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1569c24-f138-49e1-83e2-f90d2a6e433c" containerName="main" Jul 14 22:14:26.291881 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.291835 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1569c24-f138-49e1-83e2-f90d2a6e433c" containerName="main" Jul 14 22:14:26.291965 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.291907 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1569c24-f138-49e1-83e2-f90d2a6e433c" containerName="main" Jul 14 22:14:26.294827 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.294810 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.297367 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.297335 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e98c02\"/\"openshift-service-ca.crt\"" Jul 14 22:14:26.298444 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.298426 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e98c02\"/\"default-dockercfg-m8b29\"" Jul 14 22:14:26.298561 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.298462 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e98c02\"/\"sub-test-model-e98c02-kserve-self-signed-certs\"" Jul 14 22:14:26.298561 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.298462 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e98c02\"/\"kube-root-ca.crt\"" Jul 14 22:14:26.305160 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.305134 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj"] Jul 14 22:14:26.419212 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.419151 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-home\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.419212 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.419212 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef1c7613-5f77-4aa0-a808-c3b9262775c2-tls-certs\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.419455 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.419234 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-dshm\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.419455 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.419284 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgfdx\" (UniqueName: \"kubernetes.io/projected/ef1c7613-5f77-4aa0-a808-c3b9262775c2-kube-api-access-xgfdx\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.419455 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.419344 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-tmp-dir\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.419455 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.419377 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-model-cache\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.520102 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.520063 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-home\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.520311 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.520111 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef1c7613-5f77-4aa0-a808-c3b9262775c2-tls-certs\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.520311 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.520127 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-dshm\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.520311 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.520148 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgfdx\" (UniqueName: \"kubernetes.io/projected/ef1c7613-5f77-4aa0-a808-c3b9262775c2-kube-api-access-xgfdx\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.520311 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.520212 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-tmp-dir\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.520311 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.520245 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-model-cache\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.520590 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.520528 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-home\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.520647 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.520591 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-tmp-dir\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.520647 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.520613 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-model-cache\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.522427 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.522406 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-dshm\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.522698 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.522682 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef1c7613-5f77-4aa0-a808-c3b9262775c2-tls-certs\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.530255 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.530224 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgfdx\" (UniqueName: \"kubernetes.io/projected/ef1c7613-5f77-4aa0-a808-c3b9262775c2-kube-api-access-xgfdx\") pod \"sub-test-model-e98c02-kserve-6bc465dbd5-hwznj\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.604116 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.604072 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:26.631058 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.631026 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:14:26.747075 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:26.747049 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj"] Jul 14 22:14:26.749767 ip-10-0-130-223 kubenswrapper[2552]: W0714 22:14:26.749732 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef1c7613_5f77_4aa0_a808_c3b9262775c2.slice/crio-b9ac8465de0c234ca6f732cab6b8cc6b291853eca953484d8d3d8808f6806b72 WatchSource:0}: Error finding container b9ac8465de0c234ca6f732cab6b8cc6b291853eca953484d8d3d8808f6806b72: Status 404 returned error can't find the container with id b9ac8465de0c234ca6f732cab6b8cc6b291853eca953484d8d3d8808f6806b72 Jul 14 22:14:27.631577 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:27.631535 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" event={"ID":"ef1c7613-5f77-4aa0-a808-c3b9262775c2","Type":"ContainerStarted","Data":"af77ac6ee9c037a3153f2e87d19b53aff72f444c8c1bb4718c450a79f4609d09"} Jul 14 22:14:27.631981 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:27.631587 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" event={"ID":"ef1c7613-5f77-4aa0-a808-c3b9262775c2","Type":"ContainerStarted","Data":"b9ac8465de0c234ca6f732cab6b8cc6b291853eca953484d8d3d8808f6806b72"} Jul 14 22:14:27.654052 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:27.653994 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" podStartSLOduration=1.653978323 podStartE2EDuration="1.653978323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:14:27.65144967 +0000 UTC m=+2234.190155343" watchObservedRunningTime="2026-07-14 22:14:27.653978323 +0000 UTC m=+2234.192683996" Jul 14 22:14:36.604493 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:36.604398 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:36.604493 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:36.604456 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:36.617078 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:36.617052 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:36.690459 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:36.690424 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:14:53.532262 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:53.532215 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:14:53.632692 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:14:53.632650 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:18.524494 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:18.524455 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq"] Jul 14 22:15:18.547164 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:18.547133 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:19.372686 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:19.372647 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj"] Jul 14 22:15:19.688946 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:19.688851 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:21.941647 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:21.941616 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:22.074215 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:22.074159 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:24.687777 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.687744 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb"] Jul 14 22:15:24.693527 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.693055 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.696614 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.696587 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e8c0aa\"/\"rate-test-model-e8c0aa-kserve-self-signed-certs\"" Jul 14 22:15:24.701646 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.701619 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb"] Jul 14 22:15:24.712313 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.712275 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:24.816446 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.816409 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-dshm\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.816446 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.816449 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-home\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.816660 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.816487 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-tmp-dir\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.816660 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.816510 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-model-cache\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.816660 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.816537 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60a4f4a-8991-49c4-aad7-3ce924126b88-tls-certs\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.816660 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.816580 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgq8g\" (UniqueName: \"kubernetes.io/projected/a60a4f4a-8991-49c4-aad7-3ce924126b88-kube-api-access-jgq8g\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.917871 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.917826 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-dshm\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.917871 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.917871 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-home\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.918130 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.917911 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-tmp-dir\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.918130 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.917939 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-model-cache\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.918130 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.917974 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60a4f4a-8991-49c4-aad7-3ce924126b88-tls-certs\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.918130 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.918003 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgq8g\" (UniqueName: \"kubernetes.io/projected/a60a4f4a-8991-49c4-aad7-3ce924126b88-kube-api-access-jgq8g\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.918416 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.918386 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-home\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.918483 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.918419 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-tmp-dir\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.918483 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.918470 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-model-cache\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.920205 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.920168 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-dshm\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.920481 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.920463 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60a4f4a-8991-49c4-aad7-3ce924126b88-tls-certs\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:24.927400 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:24.927376 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgq8g\" (UniqueName: \"kubernetes.io/projected/a60a4f4a-8991-49c4-aad7-3ce924126b88-kube-api-access-jgq8g\") pod \"rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:25.007378 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:25.007288 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:25.142413 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:25.142382 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb"] Jul 14 22:15:25.143653 ip-10-0-130-223 kubenswrapper[2552]: W0714 22:15:25.143626 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda60a4f4a_8991_49c4_aad7_3ce924126b88.slice/crio-4d91ece77c155906ffaa923bc44e8b010b68c4d159e5288108984dc94e8ea50b WatchSource:0}: Error finding container 4d91ece77c155906ffaa923bc44e8b010b68c4d159e5288108984dc94e8ea50b: Status 404 returned error can't find the container with id 4d91ece77c155906ffaa923bc44e8b010b68c4d159e5288108984dc94e8ea50b Jul 14 22:15:25.949953 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:25.949912 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" event={"ID":"a60a4f4a-8991-49c4-aad7-3ce924126b88","Type":"ContainerStarted","Data":"bc5be217f51f7bebf397592a27e176d0d8d5a712521052fb04962b9e9c910094"} Jul 14 22:15:25.950358 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:25.949959 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" event={"ID":"a60a4f4a-8991-49c4-aad7-3ce924126b88","Type":"ContainerStarted","Data":"4d91ece77c155906ffaa923bc44e8b010b68c4d159e5288108984dc94e8ea50b"} Jul 14 22:15:25.971664 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:25.971614 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" podStartSLOduration=1.971599054 podStartE2EDuration="1.971599054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:15:25.970949976 +0000 UTC m=+2292.509655677" watchObservedRunningTime="2026-07-14 22:15:25.971599054 +0000 UTC m=+2292.510304726" Jul 14 22:15:33.545087 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.545035 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" podUID="4c782097-ec32-459f-a955-2572be3b056e" containerName="main" containerID="cri-o://60166de3e9f5fb6fb2c2f2a8846e9919b153fb25df3e2acbbff394349db066cd" gracePeriod=45 Jul 14 22:15:33.803277 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.803170 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:15:33.904277 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.904240 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-tmp-dir\") pod \"4c782097-ec32-459f-a955-2572be3b056e\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " Jul 14 22:15:33.904437 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.904288 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-home\") pod \"4c782097-ec32-459f-a955-2572be3b056e\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " Jul 14 22:15:33.904437 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.904307 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-model-cache\") pod \"4c782097-ec32-459f-a955-2572be3b056e\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " Jul 14 22:15:33.904437 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.904350 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c782097-ec32-459f-a955-2572be3b056e-tls-certs\") pod \"4c782097-ec32-459f-a955-2572be3b056e\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " Jul 14 22:15:33.904437 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.904422 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrkfq\" (UniqueName: \"kubernetes.io/projected/4c782097-ec32-459f-a955-2572be3b056e-kube-api-access-zrkfq\") pod \"4c782097-ec32-459f-a955-2572be3b056e\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " Jul 14 22:15:33.904667 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.904460 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-dshm\") pod \"4c782097-ec32-459f-a955-2572be3b056e\" (UID: \"4c782097-ec32-459f-a955-2572be3b056e\") " Jul 14 22:15:33.904667 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.904549 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c782097-ec32-459f-a955-2572be3b056e" (UID: "4c782097-ec32-459f-a955-2572be3b056e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:15:33.904667 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.904563 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-home" (OuterVolumeSpecName: "home") pod "4c782097-ec32-459f-a955-2572be3b056e" (UID: "4c782097-ec32-459f-a955-2572be3b056e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:15:33.904667 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.904632 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c782097-ec32-459f-a955-2572be3b056e" (UID: "4c782097-ec32-459f-a955-2572be3b056e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:15:33.904880 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.904789 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-tmp-dir\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:15:33.904880 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.904809 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-home\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:15:33.904880 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.904823 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-model-cache\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:15:33.906690 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.906666 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-dshm" (OuterVolumeSpecName: "dshm") pod "4c782097-ec32-459f-a955-2572be3b056e" (UID: "4c782097-ec32-459f-a955-2572be3b056e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:15:33.906690 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.906673 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c782097-ec32-459f-a955-2572be3b056e-kube-api-access-zrkfq" (OuterVolumeSpecName: "kube-api-access-zrkfq") pod "4c782097-ec32-459f-a955-2572be3b056e" (UID: "4c782097-ec32-459f-a955-2572be3b056e"). InnerVolumeSpecName "kube-api-access-zrkfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:15:33.906690 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.906665 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c782097-ec32-459f-a955-2572be3b056e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c782097-ec32-459f-a955-2572be3b056e" (UID: "4c782097-ec32-459f-a955-2572be3b056e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:15:33.998142 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.998107 2552 generic.go:356] "Generic (PLEG): container finished" podID="4c782097-ec32-459f-a955-2572be3b056e" containerID="60166de3e9f5fb6fb2c2f2a8846e9919b153fb25df3e2acbbff394349db066cd" exitCode=0 Jul 14 22:15:33.998366 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.998148 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" Jul 14 22:15:33.998366 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.998161 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" event={"ID":"4c782097-ec32-459f-a955-2572be3b056e","Type":"ContainerDied","Data":"60166de3e9f5fb6fb2c2f2a8846e9919b153fb25df3e2acbbff394349db066cd"} Jul 14 22:15:33.998366 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.998212 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq" event={"ID":"4c782097-ec32-459f-a955-2572be3b056e","Type":"ContainerDied","Data":"6b1ef72bd2593321489bd9b2214d223af80276088d7a6d433c1cafba5bde321a"} Jul 14 22:15:33.998366 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:33.998233 2552 scope.go:117] "RemoveContainer" containerID="60166de3e9f5fb6fb2c2f2a8846e9919b153fb25df3e2acbbff394349db066cd" Jul 14 22:15:34.005788 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.005764 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrkfq\" (UniqueName: \"kubernetes.io/projected/4c782097-ec32-459f-a955-2572be3b056e-kube-api-access-zrkfq\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:15:34.005890 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.005788 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c782097-ec32-459f-a955-2572be3b056e-dshm\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:15:34.005890 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.005804 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c782097-ec32-459f-a955-2572be3b056e-tls-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:15:34.008473 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.008456 2552 scope.go:117] "RemoveContainer" containerID="60166de3e9f5fb6fb2c2f2a8846e9919b153fb25df3e2acbbff394349db066cd" Jul 14 22:15:34.008744 ip-10-0-130-223 kubenswrapper[2552]: E0714 22:15:34.008724 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60166de3e9f5fb6fb2c2f2a8846e9919b153fb25df3e2acbbff394349db066cd\": container with ID starting with 60166de3e9f5fb6fb2c2f2a8846e9919b153fb25df3e2acbbff394349db066cd not found: ID does not exist" containerID="60166de3e9f5fb6fb2c2f2a8846e9919b153fb25df3e2acbbff394349db066cd" Jul 14 22:15:34.008803 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.008760 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60166de3e9f5fb6fb2c2f2a8846e9919b153fb25df3e2acbbff394349db066cd"} err="failed to get container status \"60166de3e9f5fb6fb2c2f2a8846e9919b153fb25df3e2acbbff394349db066cd\": rpc error: code = NotFound desc = could not find container \"60166de3e9f5fb6fb2c2f2a8846e9919b153fb25df3e2acbbff394349db066cd\": container with ID starting with 60166de3e9f5fb6fb2c2f2a8846e9919b153fb25df3e2acbbff394349db066cd not found: ID does not exist" Jul 14 22:15:34.019822 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.019796 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq"] Jul 14 22:15:34.025771 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.025749 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/sub-test-model-e8c0aa-kserve-696c47c87f-bzvvq"] Jul 14 22:15:34.392775 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.392703 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" podUID="ef1c7613-5f77-4aa0-a808-c3b9262775c2" containerName="main" containerID="cri-o://af77ac6ee9c037a3153f2e87d19b53aff72f444c8c1bb4718c450a79f4609d09" gracePeriod=45 Jul 14 22:15:34.639884 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.639859 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:15:34.711870 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.711770 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-dshm\") pod \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " Jul 14 22:15:34.711870 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.711821 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef1c7613-5f77-4aa0-a808-c3b9262775c2-tls-certs\") pod \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " Jul 14 22:15:34.711870 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.711842 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-tmp-dir\") pod \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " Jul 14 22:15:34.711870 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.711877 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgfdx\" (UniqueName: \"kubernetes.io/projected/ef1c7613-5f77-4aa0-a808-c3b9262775c2-kube-api-access-xgfdx\") pod \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " Jul 14 22:15:34.712266 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.711895 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-home\") pod \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " Jul 14 22:15:34.712266 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.711914 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-model-cache\") pod \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\" (UID: \"ef1c7613-5f77-4aa0-a808-c3b9262775c2\") " Jul 14 22:15:34.712266 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.712166 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-home" (OuterVolumeSpecName: "home") pod "ef1c7613-5f77-4aa0-a808-c3b9262775c2" (UID: "ef1c7613-5f77-4aa0-a808-c3b9262775c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:15:34.712266 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.712183 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef1c7613-5f77-4aa0-a808-c3b9262775c2" (UID: "ef1c7613-5f77-4aa0-a808-c3b9262775c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:15:34.712429 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.712310 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef1c7613-5f77-4aa0-a808-c3b9262775c2" (UID: "ef1c7613-5f77-4aa0-a808-c3b9262775c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:15:34.714226 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.714186 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-dshm" (OuterVolumeSpecName: "dshm") pod "ef1c7613-5f77-4aa0-a808-c3b9262775c2" (UID: "ef1c7613-5f77-4aa0-a808-c3b9262775c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:15:34.714318 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.714246 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef1c7613-5f77-4aa0-a808-c3b9262775c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef1c7613-5f77-4aa0-a808-c3b9262775c2" (UID: "ef1c7613-5f77-4aa0-a808-c3b9262775c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:15:34.714372 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.714329 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1c7613-5f77-4aa0-a808-c3b9262775c2-kube-api-access-xgfdx" (OuterVolumeSpecName: "kube-api-access-xgfdx") pod "ef1c7613-5f77-4aa0-a808-c3b9262775c2" (UID: "ef1c7613-5f77-4aa0-a808-c3b9262775c2"). InnerVolumeSpecName "kube-api-access-xgfdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:15:34.813403 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.813362 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-dshm\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:15:34.813403 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.813395 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef1c7613-5f77-4aa0-a808-c3b9262775c2-tls-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:15:34.813614 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.813427 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-tmp-dir\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:15:34.813614 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.813437 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgfdx\" (UniqueName: \"kubernetes.io/projected/ef1c7613-5f77-4aa0-a808-c3b9262775c2-kube-api-access-xgfdx\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:15:34.813614 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.813447 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-home\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:15:34.813614 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:34.813455 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef1c7613-5f77-4aa0-a808-c3b9262775c2-model-cache\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:15:35.005976 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:35.005876 2552 generic.go:356] "Generic (PLEG): container finished" podID="ef1c7613-5f77-4aa0-a808-c3b9262775c2" containerID="af77ac6ee9c037a3153f2e87d19b53aff72f444c8c1bb4718c450a79f4609d09" exitCode=0 Jul 14 22:15:35.005976 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:35.005933 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" Jul 14 22:15:35.005976 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:35.005946 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" event={"ID":"ef1c7613-5f77-4aa0-a808-c3b9262775c2","Type":"ContainerDied","Data":"af77ac6ee9c037a3153f2e87d19b53aff72f444c8c1bb4718c450a79f4609d09"} Jul 14 22:15:35.005976 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:35.005978 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj" event={"ID":"ef1c7613-5f77-4aa0-a808-c3b9262775c2","Type":"ContainerDied","Data":"b9ac8465de0c234ca6f732cab6b8cc6b291853eca953484d8d3d8808f6806b72"} Jul 14 22:15:35.006328 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:35.005993 2552 scope.go:117] "RemoveContainer" containerID="af77ac6ee9c037a3153f2e87d19b53aff72f444c8c1bb4718c450a79f4609d09" Jul 14 22:15:35.007467 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:35.007450 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:35.007559 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:35.007490 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:35.016278 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:35.016249 2552 scope.go:117] "RemoveContainer" containerID="af77ac6ee9c037a3153f2e87d19b53aff72f444c8c1bb4718c450a79f4609d09" Jul 14 22:15:35.016543 ip-10-0-130-223 kubenswrapper[2552]: E0714 22:15:35.016524 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af77ac6ee9c037a3153f2e87d19b53aff72f444c8c1bb4718c450a79f4609d09\": container with ID starting with af77ac6ee9c037a3153f2e87d19b53aff72f444c8c1bb4718c450a79f4609d09 not found: ID does not exist" containerID="af77ac6ee9c037a3153f2e87d19b53aff72f444c8c1bb4718c450a79f4609d09" Jul 14 22:15:35.016606 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:35.016552 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af77ac6ee9c037a3153f2e87d19b53aff72f444c8c1bb4718c450a79f4609d09"} err="failed to get container status \"af77ac6ee9c037a3153f2e87d19b53aff72f444c8c1bb4718c450a79f4609d09\": rpc error: code = NotFound desc = could not find container \"af77ac6ee9c037a3153f2e87d19b53aff72f444c8c1bb4718c450a79f4609d09\": container with ID starting with af77ac6ee9c037a3153f2e87d19b53aff72f444c8c1bb4718c450a79f4609d09 not found: ID does not exist" Jul 14 22:15:35.021541 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:35.021523 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:35.035359 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:35.035335 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj"] Jul 14 22:15:35.037666 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:35.037645 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e98c02/sub-test-model-e98c02-kserve-6bc465dbd5-hwznj"] Jul 14 22:15:35.414727 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:35.414682 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:35.992547 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:35.992514 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c782097-ec32-459f-a955-2572be3b056e" path="/var/lib/kubelet/pods/4c782097-ec32-459f-a955-2572be3b056e/volumes" Jul 14 22:15:35.992935 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:35.992921 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef1c7613-5f77-4aa0-a808-c3b9262775c2" path="/var/lib/kubelet/pods/ef1c7613-5f77-4aa0-a808-c3b9262775c2/volumes" Jul 14 22:15:36.023301 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:36.023272 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:15:42.950295 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:42.950259 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:55.013330 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:15:55.013295 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:11.651120 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:16:11.650979 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:11.753484 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:16:11.753443 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:19.511679 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:16:19.511642 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:19.621605 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:16:19.621568 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:52.408998 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:16:52.408959 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb"] Jul 14 22:16:52.434806 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:16:52.434775 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:53.229483 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:16:53.229448 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:54.376755 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:16:54.376708 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:54.871567 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:16:54.871527 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:55.073903 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:16:55.073864 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:55.175754 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:16:55.175646 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:55.359690 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:16:55.359658 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:59.946346 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:16:59.946297 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:07.429984 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.429917 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" podUID="a60a4f4a-8991-49c4-aad7-3ce924126b88" containerName="main" containerID="cri-o://bc5be217f51f7bebf397592a27e176d0d8d5a712521052fb04962b9e9c910094" gracePeriod=45 Jul 14 22:17:07.685757 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.685687 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:17:07.823791 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.823757 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-home\") pod \"a60a4f4a-8991-49c4-aad7-3ce924126b88\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " Jul 14 22:17:07.824010 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.823823 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-model-cache\") pod \"a60a4f4a-8991-49c4-aad7-3ce924126b88\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " Jul 14 22:17:07.824010 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.823843 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60a4f4a-8991-49c4-aad7-3ce924126b88-tls-certs\") pod \"a60a4f4a-8991-49c4-aad7-3ce924126b88\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " Jul 14 22:17:07.824010 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.823863 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-dshm\") pod \"a60a4f4a-8991-49c4-aad7-3ce924126b88\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " Jul 14 22:17:07.824010 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.823916 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-tmp-dir\") pod \"a60a4f4a-8991-49c4-aad7-3ce924126b88\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " Jul 14 22:17:07.824010 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.823933 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgq8g\" (UniqueName: \"kubernetes.io/projected/a60a4f4a-8991-49c4-aad7-3ce924126b88-kube-api-access-jgq8g\") pod \"a60a4f4a-8991-49c4-aad7-3ce924126b88\" (UID: \"a60a4f4a-8991-49c4-aad7-3ce924126b88\") " Jul 14 22:17:07.824276 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.824097 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-home" (OuterVolumeSpecName: "home") pod "a60a4f4a-8991-49c4-aad7-3ce924126b88" (UID: "a60a4f4a-8991-49c4-aad7-3ce924126b88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:17:07.824276 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.824147 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a60a4f4a-8991-49c4-aad7-3ce924126b88" (UID: "a60a4f4a-8991-49c4-aad7-3ce924126b88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:17:07.824276 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.824173 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-model-cache" (OuterVolumeSpecName: "model-cache") pod "a60a4f4a-8991-49c4-aad7-3ce924126b88" (UID: "a60a4f4a-8991-49c4-aad7-3ce924126b88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:17:07.826134 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.826107 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60a4f4a-8991-49c4-aad7-3ce924126b88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a60a4f4a-8991-49c4-aad7-3ce924126b88" (UID: "a60a4f4a-8991-49c4-aad7-3ce924126b88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:17:07.826264 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.826134 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-dshm" (OuterVolumeSpecName: "dshm") pod "a60a4f4a-8991-49c4-aad7-3ce924126b88" (UID: "a60a4f4a-8991-49c4-aad7-3ce924126b88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:17:07.826264 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.826148 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60a4f4a-8991-49c4-aad7-3ce924126b88-kube-api-access-jgq8g" (OuterVolumeSpecName: "kube-api-access-jgq8g") pod "a60a4f4a-8991-49c4-aad7-3ce924126b88" (UID: "a60a4f4a-8991-49c4-aad7-3ce924126b88"). InnerVolumeSpecName "kube-api-access-jgq8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:17:07.924926 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.924886 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-model-cache\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:17:07.924926 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.924918 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60a4f4a-8991-49c4-aad7-3ce924126b88-tls-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:17:07.924926 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.924927 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-dshm\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:17:07.924926 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.924935 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-tmp-dir\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:17:07.925239 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.924945 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgq8g\" (UniqueName: \"kubernetes.io/projected/a60a4f4a-8991-49c4-aad7-3ce924126b88-kube-api-access-jgq8g\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:17:07.925239 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:07.924955 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60a4f4a-8991-49c4-aad7-3ce924126b88-home\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\"" Jul 14 22:17:08.513130 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:08.513094 2552 generic.go:356] "Generic (PLEG): container finished" podID="a60a4f4a-8991-49c4-aad7-3ce924126b88" containerID="bc5be217f51f7bebf397592a27e176d0d8d5a712521052fb04962b9e9c910094" exitCode=0 Jul 14 22:17:08.513602 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:08.513142 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" Jul 14 22:17:08.513602 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:08.513172 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" event={"ID":"a60a4f4a-8991-49c4-aad7-3ce924126b88","Type":"ContainerDied","Data":"bc5be217f51f7bebf397592a27e176d0d8d5a712521052fb04962b9e9c910094"} Jul 14 22:17:08.513602 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:08.513236 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb" event={"ID":"a60a4f4a-8991-49c4-aad7-3ce924126b88","Type":"ContainerDied","Data":"4d91ece77c155906ffaa923bc44e8b010b68c4d159e5288108984dc94e8ea50b"} Jul 14 22:17:08.513602 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:08.513257 2552 scope.go:117] "RemoveContainer" containerID="bc5be217f51f7bebf397592a27e176d0d8d5a712521052fb04962b9e9c910094" Jul 14 22:17:08.522762 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:08.522745 2552 scope.go:117] "RemoveContainer" containerID="bc5be217f51f7bebf397592a27e176d0d8d5a712521052fb04962b9e9c910094" Jul 14 22:17:08.523003 ip-10-0-130-223 kubenswrapper[2552]: E0714 22:17:08.522981 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5be217f51f7bebf397592a27e176d0d8d5a712521052fb04962b9e9c910094\": container with ID starting with bc5be217f51f7bebf397592a27e176d0d8d5a712521052fb04962b9e9c910094 not found: ID does not exist" containerID="bc5be217f51f7bebf397592a27e176d0d8d5a712521052fb04962b9e9c910094" Jul 14 22:17:08.523069 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:08.523016 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5be217f51f7bebf397592a27e176d0d8d5a712521052fb04962b9e9c910094"} err="failed to get container status \"bc5be217f51f7bebf397592a27e176d0d8d5a712521052fb04962b9e9c910094\": rpc error: code = NotFound desc = could not find container \"bc5be217f51f7bebf397592a27e176d0d8d5a712521052fb04962b9e9c910094\": container with ID starting with bc5be217f51f7bebf397592a27e176d0d8d5a712521052fb04962b9e9c910094 not found: ID does not exist" Jul 14 22:17:08.533477 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:08.533452 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb"] Jul 14 22:17:08.537988 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:08.537966 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e8c0aa/rate-test-model-e8c0aa-kserve-7c98c57ddc-w4gkb"] Jul 14 22:17:09.991857 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:09.991820 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a60a4f4a-8991-49c4-aad7-3ce924126b88" path="/var/lib/kubelet/pods/a60a4f4a-8991-49c4-aad7-3ce924126b88/volumes" Jul 14 22:17:14.154808 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:14.154779 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 22:17:14.162058 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:14.162033 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 22:17:14.163717 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:14.163697 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 22:17:14.170001 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:14.169984 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fxphv_9cc0cd22-ce64-4aaf-8af0-2618da80d640/ovn-acl-logging/0.log" Jul 14 22:17:37.861035 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:37.860944 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:38.109763 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:38.109732 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:38.374338 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:38.374297 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:38.548969 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:38.548931 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:39.048938 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:39.048901 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:39.149931 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:39.149897 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:39.337881 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:39.337853 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:45.799335 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:45.799302 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:45.870251 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:45.870185 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:46.097218 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:46.097170 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:46.339291 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:46.339258 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:46.812850 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:46.812812 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7758664dd6-hzmr5_b2cf3e3a-7a0e-4b7d-8fa9-3a485497aa21/authorino/0.log" Jul 14 22:17:52.532419 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:52.532386 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-wnrf5_6cd94038-ea1c-4009-8e2c-838e8b43f4b5/server/0.log" Jul 14 22:17:52.662666 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:52.662637 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-c45n9_03b03461-030e-4ae8-9292-f85650d0c18a/manager/0.log" Jul 14 22:17:53.057013 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:53.056984 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c55cf7c4c-zkxmh_1319bf20-cc7d-474e-bd56-89e5dee487b7/copy-manifests/0.log" Jul 14 22:17:53.065069 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:53.065036 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c55cf7c4c-zkxmh_1319bf20-cc7d-474e-bd56-89e5dee487b7/manager/0.log" Jul 14 22:17:54.027095 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:54.027061 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9_3446180e-7b30-45e7-a5e9-118a1384d1e7/util/0.log" Jul 14 22:17:54.034357 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:54.034331 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9_3446180e-7b30-45e7-a5e9-118a1384d1e7/pull/0.log" Jul 14 22:17:54.041646 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:54.041623 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9_3446180e-7b30-45e7-a5e9-118a1384d1e7/extract/0.log" Jul 14 22:17:54.169314 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:54.169289 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g_c63ce66e-682c-4e21-ac5b-b7d748d66190/util/0.log" Jul 14 22:17:54.176350 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:54.176325 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g_c63ce66e-682c-4e21-ac5b-b7d748d66190/pull/0.log" Jul 14 22:17:54.183026 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:54.183005 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g_c63ce66e-682c-4e21-ac5b-b7d748d66190/extract/0.log" Jul 14 22:17:54.316439 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:54.316344 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx_00780a1c-747c-41fa-a237-16fd726b14d3/util/0.log" Jul 14 22:17:54.327081 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:54.327053 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx_00780a1c-747c-41fa-a237-16fd726b14d3/pull/0.log" Jul 14 22:17:54.340939 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:54.340911 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx_00780a1c-747c-41fa-a237-16fd726b14d3/extract/0.log" Jul 14 22:17:54.465452 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:54.465424 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v_d93ce281-4e29-4317-a841-8f530d9dc34d/util/0.log" Jul 14 22:17:54.473168 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:54.473136 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v_d93ce281-4e29-4317-a841-8f530d9dc34d/pull/0.log" Jul 14 22:17:54.480594 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:54.480568 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v_d93ce281-4e29-4317-a841-8f530d9dc34d/extract/0.log" Jul 14 22:17:54.616028 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:54.615997 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7758664dd6-hzmr5_b2cf3e3a-7a0e-4b7d-8fa9-3a485497aa21/authorino/0.log" Jul 14 22:17:55.004522 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:55.004440 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zfz7t_e30d8d46-d5e4-4a96-b261-8a2130c887ac/kuadrant-console-plugin/0.log" Jul 14 22:17:56.154014 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:56.153980 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65594b5d8c-fbm2h_ea2769b0-323f-458f-a947-88511ea44ee1/kube-auth-proxy/0.log" Jul 14 22:17:56.671516 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:56.671483 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-zkj4p_14003eef-8c35-409d-b56d-62d729ad89d0/payload-processing/0.log" Jul 14 22:17:56.793550 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:56.793517 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c67f54db8-mlqfz_b1ed9b70-c79d-44a2-950c-af1d0b6bdd57/router/0.log" Jul 14 22:17:57.284868 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:57.284841 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-ngcmn_7af7a3a1-bcfb-4ceb-b66a-ef97b4aa976e/main/0.log" Jul 14 22:17:57.658461 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:57.658421 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-zpmnh_c12db4e4-83ee-4771-a746-94449786b2ea/main/0.log" Jul 14 22:17:57.779409 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:17:57.779376 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-5zdmv_a8eeedcb-c169-4765-9ea5-ae70eb757fa9/main/0.log" Jul 14 22:18:02.120310 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.120273 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7zq7r/must-gather-jwd4t"] Jul 14 22:18:02.120727 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.120644 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c782097-ec32-459f-a955-2572be3b056e" containerName="main" Jul 14 22:18:02.120727 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.120655 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c782097-ec32-459f-a955-2572be3b056e" containerName="main" Jul 14 22:18:02.120727 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.120678 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef1c7613-5f77-4aa0-a808-c3b9262775c2" containerName="main" Jul 14 22:18:02.120727 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.120686 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef1c7613-5f77-4aa0-a808-c3b9262775c2" containerName="main" Jul 14 22:18:02.120727 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.120694 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60a4f4a-8991-49c4-aad7-3ce924126b88" containerName="main" Jul 14 22:18:02.120727 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.120700 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60a4f4a-8991-49c4-aad7-3ce924126b88" containerName="main" Jul 14 22:18:02.120941 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.120765 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c782097-ec32-459f-a955-2572be3b056e" containerName="main" Jul 14 22:18:02.120941 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.120774 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef1c7613-5f77-4aa0-a808-c3b9262775c2" containerName="main" Jul 14 22:18:02.120941 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.120782 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60a4f4a-8991-49c4-aad7-3ce924126b88" containerName="main" Jul 14 22:18:02.124042 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.124024 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7zq7r/must-gather-jwd4t" Jul 14 22:18:02.126308 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.126278 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:02.126830 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.126811 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7zq7r\"/\"kube-root-ca.crt\"" Jul 14 22:18:02.127784 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.127767 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7zq7r\"/\"default-dockercfg-t7ph4\"" Jul 14 22:18:02.127879 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.127768 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7zq7r\"/\"openshift-service-ca.crt\"" Jul 14 22:18:02.141067 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.141037 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7zq7r/must-gather-jwd4t"] Jul 14 22:18:02.196372 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.196332 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:02.211287 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.211254 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/52c24740-dae5-42c2-b860-2e9aebe58a79-must-gather-output\") pod \"must-gather-jwd4t\" (UID: \"52c24740-dae5-42c2-b860-2e9aebe58a79\") " pod="openshift-must-gather-7zq7r/must-gather-jwd4t" Jul 14 22:18:02.211287 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.211286 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlr4k\" (UniqueName: \"kubernetes.io/projected/52c24740-dae5-42c2-b860-2e9aebe58a79-kube-api-access-qlr4k\") pod \"must-gather-jwd4t\" (UID: \"52c24740-dae5-42c2-b860-2e9aebe58a79\") " pod="openshift-must-gather-7zq7r/must-gather-jwd4t" Jul 14 22:18:02.312358 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.312262 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/52c24740-dae5-42c2-b860-2e9aebe58a79-must-gather-output\") pod \"must-gather-jwd4t\" (UID: \"52c24740-dae5-42c2-b860-2e9aebe58a79\") " pod="openshift-must-gather-7zq7r/must-gather-jwd4t" Jul 14 22:18:02.312358 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.312299 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlr4k\" (UniqueName: \"kubernetes.io/projected/52c24740-dae5-42c2-b860-2e9aebe58a79-kube-api-access-qlr4k\") pod \"must-gather-jwd4t\" (UID: \"52c24740-dae5-42c2-b860-2e9aebe58a79\") " pod="openshift-must-gather-7zq7r/must-gather-jwd4t" Jul 14 22:18:02.312621 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.312600 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/52c24740-dae5-42c2-b860-2e9aebe58a79-must-gather-output\") pod \"must-gather-jwd4t\" (UID: \"52c24740-dae5-42c2-b860-2e9aebe58a79\") " pod="openshift-must-gather-7zq7r/must-gather-jwd4t" Jul 14 22:18:02.326863 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.326838 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlr4k\" (UniqueName: \"kubernetes.io/projected/52c24740-dae5-42c2-b860-2e9aebe58a79-kube-api-access-qlr4k\") pod \"must-gather-jwd4t\" (UID: \"52c24740-dae5-42c2-b860-2e9aebe58a79\") " pod="openshift-must-gather-7zq7r/must-gather-jwd4t" Jul 14 22:18:02.433834 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.433791 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7zq7r/must-gather-jwd4t" Jul 14 22:18:02.586879 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.586793 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7zq7r/must-gather-jwd4t"] Jul 14 22:18:02.590072 ip-10-0-130-223 kubenswrapper[2552]: W0714 22:18:02.590045 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52c24740_dae5_42c2_b860_2e9aebe58a79.slice/crio-0092ca9b0706759a160b0878f7d9f5479c7c38f5f8371a51145569139942d799 WatchSource:0}: Error finding container 0092ca9b0706759a160b0878f7d9f5479c7c38f5f8371a51145569139942d799: Status 404 returned error can't find the container with id 0092ca9b0706759a160b0878f7d9f5479c7c38f5f8371a51145569139942d799 Jul 14 22:18:02.592068 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.592050 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 22:18:02.803648 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:02.803612 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7zq7r/must-gather-jwd4t" event={"ID":"52c24740-dae5-42c2-b860-2e9aebe58a79","Type":"ContainerStarted","Data":"0092ca9b0706759a160b0878f7d9f5479c7c38f5f8371a51145569139942d799"} Jul 14 22:18:03.812883 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:03.812778 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7zq7r/must-gather-jwd4t" event={"ID":"52c24740-dae5-42c2-b860-2e9aebe58a79","Type":"ContainerStarted","Data":"1cb431399153650f04c67b876b1b1c531f006d22bc44ae548807c2edb6cd054e"} Jul 14 22:18:03.812883 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:03.812834 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7zq7r/must-gather-jwd4t" event={"ID":"52c24740-dae5-42c2-b860-2e9aebe58a79","Type":"ContainerStarted","Data":"e19d8bbb3133eb25d3405430e99728424e448f164c2c2a6328d26156676ecba6"} Jul 14 22:18:03.833556 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:03.833484 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7zq7r/must-gather-jwd4t" podStartSLOduration=0.886557194 podStartE2EDuration="1.833460341s" totalImagesPullingTime="946.903147ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:18:03.83116074 +0000 UTC m=+2450.369866413" watchObservedRunningTime="2026-07-14 22:18:03.833460341 +0000 UTC m=+2450.372166014" Jul 14 22:18:05.214711 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:05.214667 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f2b5s_d0336426-165c-4987-beaa-f74fe715d480/global-pull-secret-syncer/0.log" Jul 14 22:18:05.374020 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:05.373986 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9thwv_451155f1-ad03-4be3-b198-895002675826/konnectivity-agent/0.log" Jul 14 22:18:05.497753 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:05.497672 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-223.ec2.internal_6347ca5f52a32e5fefe18e2f2e47b4f4/haproxy/0.log" Jul 14 22:18:09.976046 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:09.975934 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9_3446180e-7b30-45e7-a5e9-118a1384d1e7/extract/0.log" Jul 14 22:18:10.004912 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:10.004886 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9_3446180e-7b30-45e7-a5e9-118a1384d1e7/util/0.log" Jul 14 22:18:10.034479 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:10.034449 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cqfd9_3446180e-7b30-45e7-a5e9-118a1384d1e7/pull/0.log" Jul 14 22:18:10.068680 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:10.068645 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g_c63ce66e-682c-4e21-ac5b-b7d748d66190/extract/0.log" Jul 14 22:18:10.097924 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:10.097883 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g_c63ce66e-682c-4e21-ac5b-b7d748d66190/util/0.log" Jul 14 22:18:10.123309 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:10.123276 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p7n5g_c63ce66e-682c-4e21-ac5b-b7d748d66190/pull/0.log" Jul 14 22:18:10.156449 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:10.156412 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx_00780a1c-747c-41fa-a237-16fd726b14d3/extract/0.log" Jul 14 22:18:10.181268 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:10.181242 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx_00780a1c-747c-41fa-a237-16fd726b14d3/util/0.log" Jul 14 22:18:10.206454 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:10.206426 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73znkgx_00780a1c-747c-41fa-a237-16fd726b14d3/pull/0.log" Jul 14 22:18:10.242463 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:10.242366 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v_d93ce281-4e29-4317-a841-8f530d9dc34d/extract/0.log" Jul 14 22:18:10.270712 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:10.270669 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v_d93ce281-4e29-4317-a841-8f530d9dc34d/util/0.log" Jul 14 22:18:10.296751 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:10.296707 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhq4v_d93ce281-4e29-4317-a841-8f530d9dc34d/pull/0.log" Jul 14 22:18:10.763211 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:10.762408 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7758664dd6-hzmr5_b2cf3e3a-7a0e-4b7d-8fa9-3a485497aa21/authorino/0.log" Jul 14 22:18:10.880096 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:10.880059 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zfz7t_e30d8d46-d5e4-4a96-b261-8a2130c887ac/kuadrant-console-plugin/0.log" Jul 14 22:18:13.209127 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:13.209086 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-82wc4_587d0871-09d8-4223-955c-52d205e0051b/cluster-monitoring-operator/0.log" Jul 14 22:18:13.466372 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:13.466292 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jlt6n_175e5d82-2c32-4067-90be-cccc501c0990/node-exporter/0.log" Jul 14 22:18:13.493796 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:13.493765 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jlt6n_175e5d82-2c32-4067-90be-cccc501c0990/kube-rbac-proxy/0.log" Jul 14 22:18:13.520243 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:13.520216 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jlt6n_175e5d82-2c32-4067-90be-cccc501c0990/init-textfile/0.log" Jul 14 22:18:15.457838 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.457795 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6"] Jul 14 22:18:15.465334 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.465299 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-8cggq_c03004c3-29b8-4813-85e1-5df04bd12a34/networking-console-plugin/0.log" Jul 14 22:18:15.465913 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.465887 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:15.474535 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.474267 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6"] Jul 14 22:18:15.562741 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.562691 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e227981-f621-4b50-b61d-6763264c18a7-sys\") pod \"perf-node-gather-daemonset-xb7f6\" (UID: \"9e227981-f621-4b50-b61d-6763264c18a7\") " pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:15.562955 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.562775 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e227981-f621-4b50-b61d-6763264c18a7-lib-modules\") pod \"perf-node-gather-daemonset-xb7f6\" (UID: \"9e227981-f621-4b50-b61d-6763264c18a7\") " pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:15.562955 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.562802 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvnxp\" (UniqueName: \"kubernetes.io/projected/9e227981-f621-4b50-b61d-6763264c18a7-kube-api-access-qvnxp\") pod \"perf-node-gather-daemonset-xb7f6\" (UID: \"9e227981-f621-4b50-b61d-6763264c18a7\") " pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:15.562955 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.562882 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e227981-f621-4b50-b61d-6763264c18a7-proc\") pod \"perf-node-gather-daemonset-xb7f6\" (UID: \"9e227981-f621-4b50-b61d-6763264c18a7\") " pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:15.562955 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.562918 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e227981-f621-4b50-b61d-6763264c18a7-podres\") pod \"perf-node-gather-daemonset-xb7f6\" (UID: \"9e227981-f621-4b50-b61d-6763264c18a7\") " pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:15.663889 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.663841 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e227981-f621-4b50-b61d-6763264c18a7-lib-modules\") pod \"perf-node-gather-daemonset-xb7f6\" (UID: \"9e227981-f621-4b50-b61d-6763264c18a7\") " pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:15.664092 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.663898 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvnxp\" (UniqueName: \"kubernetes.io/projected/9e227981-f621-4b50-b61d-6763264c18a7-kube-api-access-qvnxp\") pod \"perf-node-gather-daemonset-xb7f6\" (UID: \"9e227981-f621-4b50-b61d-6763264c18a7\") " pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:15.664092 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.663984 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e227981-f621-4b50-b61d-6763264c18a7-proc\") pod \"perf-node-gather-daemonset-xb7f6\" (UID: \"9e227981-f621-4b50-b61d-6763264c18a7\") " pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:15.664092 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.664019 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e227981-f621-4b50-b61d-6763264c18a7-podres\") pod \"perf-node-gather-daemonset-xb7f6\" (UID: \"9e227981-f621-4b50-b61d-6763264c18a7\") " pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:15.664092 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.664064 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e227981-f621-4b50-b61d-6763264c18a7-sys\") pod \"perf-node-gather-daemonset-xb7f6\" (UID: \"9e227981-f621-4b50-b61d-6763264c18a7\") " pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:15.664329 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.664173 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e227981-f621-4b50-b61d-6763264c18a7-sys\") pod \"perf-node-gather-daemonset-xb7f6\" (UID: \"9e227981-f621-4b50-b61d-6763264c18a7\") " pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:15.664391 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.664341 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e227981-f621-4b50-b61d-6763264c18a7-lib-modules\") pod \"perf-node-gather-daemonset-xb7f6\" (UID: \"9e227981-f621-4b50-b61d-6763264c18a7\") " pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:15.664740 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.664705 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e227981-f621-4b50-b61d-6763264c18a7-proc\") pod \"perf-node-gather-daemonset-xb7f6\" (UID: \"9e227981-f621-4b50-b61d-6763264c18a7\") " pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:15.664853 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.664833 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e227981-f621-4b50-b61d-6763264c18a7-podres\") pod \"perf-node-gather-daemonset-xb7f6\" (UID: \"9e227981-f621-4b50-b61d-6763264c18a7\") " pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:15.674637 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.674608 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvnxp\" (UniqueName: \"kubernetes.io/projected/9e227981-f621-4b50-b61d-6763264c18a7-kube-api-access-qvnxp\") pod \"perf-node-gather-daemonset-xb7f6\" (UID: \"9e227981-f621-4b50-b61d-6763264c18a7\") " pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:15.779890 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.779788 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:15.981053 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:15.980574 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6"] Jul 14 22:18:15.983218 ip-10-0-130-223 kubenswrapper[2552]: W0714 22:18:15.983166 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9e227981_f621_4b50_b61d_6763264c18a7.slice/crio-ee780b9157b906b7332394109bb4e3492062794ea78bd966c16621d78e72c1f9 WatchSource:0}: Error finding container ee780b9157b906b7332394109bb4e3492062794ea78bd966c16621d78e72c1f9: Status 404 returned error can't find the container with id ee780b9157b906b7332394109bb4e3492062794ea78bd966c16621d78e72c1f9 Jul 14 22:18:16.166430 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:16.166386 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/1.log" Jul 14 22:18:16.176293 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:16.176258 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x65n5_a6cb0783-1901-4441-a543-215ede6cef78/console-operator/2.log" Jul 14 22:18:16.696852 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:16.696810 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-xrqvn_087cfd76-4d93-43b7-b145-9915a91d00fd/download-server/0.log" Jul 14 22:18:16.941751 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:16.941714 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" event={"ID":"9e227981-f621-4b50-b61d-6763264c18a7","Type":"ContainerStarted","Data":"8e0882f8a9fefd77b802b9f5a007ba91fd05bc71a36b4bcce7d93872658d7b09"} Jul 14 22:18:16.941960 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:16.941778 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" event={"ID":"9e227981-f621-4b50-b61d-6763264c18a7","Type":"ContainerStarted","Data":"ee780b9157b906b7332394109bb4e3492062794ea78bd966c16621d78e72c1f9"} Jul 14 22:18:16.941960 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:16.941822 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:16.962538 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:16.962431 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" podStartSLOduration=1.962409364 podStartE2EDuration="1.962409364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:18:16.960738196 +0000 UTC m=+2463.499443852" watchObservedRunningTime="2026-07-14 22:18:16.962409364 +0000 UTC m=+2463.501115037" Jul 14 22:18:17.353133 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:17.353078 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-cdl6v_b7676ddd-9ceb-4328-98a2-44172842a0d8/volume-data-source-validator/0.log" Jul 14 22:18:18.567980 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:18.567946 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jrvlv_4e461f5b-ff8f-4018-a3d2-9de160cabffd/dns/0.log" Jul 14 22:18:18.595713 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:18.595683 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jrvlv_4e461f5b-ff8f-4018-a3d2-9de160cabffd/kube-rbac-proxy/0.log" Jul 14 22:18:18.652265 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:18.652237 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c4vgt_ba716d4b-af3d-44a4-a7d0-d134e8c805f3/dns-node-resolver/0.log" Jul 14 22:18:19.301464 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:19.301429 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2545q_b59fc23b-4e0f-4e2a-8cc7-2bd145c4ab10/node-ca/0.log" Jul 14 22:18:20.521951 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:20.521916 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65594b5d8c-fbm2h_ea2769b0-323f-458f-a947-88511ea44ee1/kube-auth-proxy/0.log" Jul 14 22:18:20.719757 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:20.719708 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-zkj4p_14003eef-8c35-409d-b56d-62d729ad89d0/payload-processing/0.log" Jul 14 22:18:20.750674 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:20.750644 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c67f54db8-mlqfz_b1ed9b70-c79d-44a2-950c-af1d0b6bdd57/router/0.log" Jul 14 22:18:21.542704 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:21.542670 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xxhmt_9687cbd1-86bc-4429-9f4d-a8cda7399acb/serve-healthcheck-canary/0.log" Jul 14 22:18:22.081310 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:22.081273 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-sgt5s_eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87/insights-operator/0.log" Jul 14 22:18:22.081704 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:22.081675 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-sgt5s_eb1713bc-74d9-4b7f-9d0e-c3450d0c6c87/insights-operator/1.log" Jul 14 22:18:22.263024 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:22.262993 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pcscn_6c40e5ea-9442-4e32-8636-999f09c6cee6/kube-rbac-proxy/0.log" Jul 14 22:18:22.288858 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:22.288821 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pcscn_6c40e5ea-9442-4e32-8636-999f09c6cee6/exporter/0.log" Jul 14 22:18:22.315867 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:22.315828 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pcscn_6c40e5ea-9442-4e32-8636-999f09c6cee6/extractor/0.log" Jul 14 22:18:22.960451 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:22.960424 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7zq7r/perf-node-gather-daemonset-xb7f6" Jul 14 22:18:25.304898 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:25.304807 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-wnrf5_6cd94038-ea1c-4009-8e2c-838e8b43f4b5/server/0.log" Jul 14 22:18:25.428600 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:25.428564 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-c45n9_03b03461-030e-4ae8-9292-f85650d0c18a/manager/0.log" Jul 14 22:18:25.614963 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:25.614920 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c55cf7c4c-zkxmh_1319bf20-cc7d-474e-bd56-89e5dee487b7/manager/0.log" Jul 14 22:18:25.641081 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:25.640972 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c55cf7c4c-zkxmh_1319bf20-cc7d-474e-bd56-89e5dee487b7/copy-manifests/0.log" Jul 14 22:18:26.347422 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:26.347376 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65594b5d8c-fbm2h_ea2769b0-323f-458f-a947-88511ea44ee1/kube-auth-proxy/0.log" Jul 14 22:18:26.506104 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:26.506070 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-zkj4p_14003eef-8c35-409d-b56d-62d729ad89d0/payload-processing/0.log" Jul 14 22:18:26.521843 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:26.521806 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c67f54db8-mlqfz_b1ed9b70-c79d-44a2-950c-af1d0b6bdd57/router/0.log" Jul 14 22:18:27.122244 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:27.122096 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-65d6b5d88-8q7q6_025a34dc-2764-4ca3-a424-53b872ced886/manager/0.log" Jul 14 22:18:27.853841 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:27.853808 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65594b5d8c-fbm2h_ea2769b0-323f-458f-a947-88511ea44ee1/kube-auth-proxy/0.log" Jul 14 22:18:28.018262 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:28.018164 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-zkj4p_14003eef-8c35-409d-b56d-62d729ad89d0/payload-processing/0.log" Jul 14 22:18:28.035555 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:28.035521 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c67f54db8-mlqfz_b1ed9b70-c79d-44a2-950c-af1d0b6bdd57/router/0.log" Jul 14 22:18:32.790365 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:32.790274 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-77p64_c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3/kube-storage-version-migrator-operator/1.log" Jul 14 22:18:32.792244 ip-10-0-130-223 kubenswrapper[2552]: I0714 22:18:32.792177 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-77p64_c9ca6103-e1eb-4d72-9bcc-444d8f8cedc3/kube-storage-version-migrator-operator/0.log"